[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:31 dist-test-slave-01nr krb5kdc[27726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:31 dist-test-slave-01nr krb5kdc[27726](info): set up 2 sockets
Apr 29 19:59:31 dist-test-slave-01nr krb5kdc[27726](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260429 19:59:32.968497 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T19:59:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T19:59:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:33.115499 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37337
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:37337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:33.225642 27742 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:33.225888 27742 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:33.225943 27742 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:33.229395 27742 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 19:59:33.229470 27742 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:33.229494 27742 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:33.229514 27742 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 19:59:33.229532 27742 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 19:59:33.234047 27742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:37337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:33.235194 27742 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:33.236016 27742 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:33.241703 27747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:33.241688 27750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:33.241691 27748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:33.241890 27742 server_base.cc:1061] running on GCE node
I20260429 19:59:33.242482 27742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:33.243474 27742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:33.244653 27742 hybrid_clock.cc:648] HybridClock initialized: now 1777492773244584 us; error 84 us; skew 500 ppm
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:33.247476 27742 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 19:59:33.248515 27742 webserver.cc:492] Webserver started at http://127.27.17.254:42735/ using document root <none> and password file <none>
I20260429 19:59:33.249075 27742 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:33.249151 27742 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:33.249356 27742 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:33.251085 27742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "d987d7d4d55f4e06b63c6422e703c72a"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "70b5f3ecddd29f5a7d7c121b5225d2e7"
server_key_iv: "152e60c00bc748515166d01bdc2f82e7"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.251579 27742 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "d987d7d4d55f4e06b63c6422e703c72a"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "70b5f3ecddd29f5a7d7c121b5225d2e7"
server_key_iv: "152e60c00bc748515166d01bdc2f82e7"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.254997 27742 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 19:59:33.257166 27757 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.258143 27742 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260429 19:59:33.258285 27742 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "d987d7d4d55f4e06b63c6422e703c72a"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "70b5f3ecddd29f5a7d7c121b5225d2e7"
server_key_iv: "152e60c00bc748515166d01bdc2f82e7"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.258394 27742 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:33.274749 27742 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:33.277621 27742 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:33.277820 27742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:33.285697 27742 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:37337
I20260429 19:59:33.285705 27809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:37337 every 8 connection(s)
I20260429 19:59:33.286731 27742 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 19:59:33.289566 27810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:33.291695 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 27742
I20260429 19:59:33.291851 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 19:59:33.293215 27719 external_mini_cluster.cc:1468] Setting key 5a9fd9c6f7f8b57057563831780ff8cd
I20260429 19:59:33.296236 27810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a: Bootstrap starting.
I20260429 19:59:33.298771 27810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:33.299726 27810 log.cc:826] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:33.301822 27810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a: No bootstrap required, opened a new log
I20260429 19:59:33.305015 27810 raft_consensus.cc:359] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d987d7d4d55f4e06b63c6422e703c72a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37337 } }
I20260429 19:59:33.305262 27810 raft_consensus.cc:385] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:33.305361 27810 raft_consensus.cc:740] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d987d7d4d55f4e06b63c6422e703c72a, State: Initialized, Role: FOLLOWER
I20260429 19:59:33.305800 27810 consensus_queue.cc:260] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d987d7d4d55f4e06b63c6422e703c72a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37337 } }
I20260429 19:59:33.305968 27810 raft_consensus.cc:399] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 19:59:33.306056 27810 raft_consensus.cc:493] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 19:59:33.306169 27810 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 0 FOLLOWER]: Advancing to term 1
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:33.307451 27810 raft_consensus.cc:515] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d987d7d4d55f4e06b63c6422e703c72a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37337 } }
I20260429 19:59:33.307865 27810 leader_election.cc:304] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d987d7d4d55f4e06b63c6422e703c72a; no voters:
I20260429 19:59:33.308260 27810 leader_election.cc:290] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 19:59:33.308506 27815 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:33.308750 27815 raft_consensus.cc:697] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [term 1 LEADER]: Becoming Leader. State: Replica: d987d7d4d55f4e06b63c6422e703c72a, State: Running, Role: LEADER
I20260429 19:59:33.309136 27815 consensus_queue.cc:237] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d987d7d4d55f4e06b63c6422e703c72a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37337 } }
I20260429 19:59:33.309731 27810 sys_catalog.cc:565] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [sys.catalog]: configured and running, proceeding with master startup.
I20260429 19:59:33.310992 27816 sys_catalog.cc:455] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d987d7d4d55f4e06b63c6422e703c72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d987d7d4d55f4e06b63c6422e703c72a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37337 } } }
I20260429 19:59:33.311124 27816 sys_catalog.cc:458] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:33.311280 27817 sys_catalog.cc:455] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d987d7d4d55f4e06b63c6422e703c72a. Latest consensus state: current_term: 1 leader_uuid: "d987d7d4d55f4e06b63c6422e703c72a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d987d7d4d55f4e06b63c6422e703c72a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37337 } } }
I20260429 19:59:33.311350 27817 sys_catalog.cc:458] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:33.311590 27824 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 19:59:33.315009 27824 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 19:59:33.315697 27813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:33.295883 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.27.17.254:37337)
0429 19:59:33.296344 (+ 461us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:33.296358 (+ 14us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:33.296856 (+ 498us) server_negotiation.cc:408] Connection header received
0429 19:59:33.299246 (+ 2390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:33.299273 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:33.299651 (+ 378us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:33.299982 (+ 331us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:33.301754 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.302883 (+ 1129us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:33.303604 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.303881 (+ 277us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:33.307750 (+ 3869us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:33.307779 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:33.307795 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:33.307836 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:33.311010 (+ 3174us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:33.311759 (+ 749us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:33.311766 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:33.311771 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:33.311870 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:33.312269 (+ 399us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:33.312272 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:33.312274 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:33.312692 (+ 418us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:33.312867 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:33.314527 (+ 1660us) server_negotiation.cc:300] Negotiation successful
0429 19:59:33.314728 (+ 201us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":137,"threads_started":1}
I20260429 19:59:33.321153 27824 catalog_manager.cc:1357] Generated new cluster ID: 3fb1b98e836544388acd38a79269f8a4
I20260429 19:59:33.321278 27824 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 19:59:33.328819 27824 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 19:59:33.329998 27824 catalog_manager.cc:1514] Loading token signing keys...
I20260429 19:59:33.344383 27824 catalog_manager.cc:6044] T 00000000000000000000000000000000 P d987d7d4d55f4e06b63c6422e703c72a: Generated new TSK 0
I20260429 19:59:33.345073 27824 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:33.406986 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37337
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:33.518421 27838 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:33.518724 27838 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:33.518821 27838 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:33.522396 27838 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:33.522486 27838 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:33.522598 27838 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 19:59:33.527179 27838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:33.528357 27838 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:33.529292 27838 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:33.536098 27843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:33.536157 27838 server_base.cc:1061] running on GCE node
W20260429 19:59:33.536170 27846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:33.536098 27844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:33.536828 27838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:33.537410 27838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:33.538617 27838 hybrid_clock.cc:648] HybridClock initialized: now 1777492773538593 us; error 44 us; skew 500 ppm
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:33.541504 27838 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 19:59:33.542657 27838 webserver.cc:492] Webserver started at http://127.27.17.193:39061/ using document root <none> and password file <none>
I20260429 19:59:33.543285 27838 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:33.543358 27838 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:33.543562 27838 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:33.545369 27838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "3a439b7dd4804c41809dda7203ac4f00"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "34b3486f060afafc8dd734e4cbef7c67"
server_key_iv: "4c44f2a0d7bc697a6af03dd692408437"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.545863 27838 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "3a439b7dd4804c41809dda7203ac4f00"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "34b3486f060afafc8dd734e4cbef7c67"
server_key_iv: "4c44f2a0d7bc697a6af03dd692408437"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.549248 27838 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260429 19:59:33.551481 27853 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.552495 27838 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 19:59:33.552644 27838 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "3a439b7dd4804c41809dda7203ac4f00"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "34b3486f060afafc8dd734e4cbef7c67"
server_key_iv: "4c44f2a0d7bc697a6af03dd692408437"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.552759 27838 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:33.573068 27838 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:33.575860 27838 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:33.576076 27838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:33.576663 27838 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:33.577587 27838 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:33.577662 27838 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.577739 27838 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:33.577782 27838 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.588143 27838 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:44341
I20260429 19:59:33.588160 27966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:44341 every 8 connection(s)
I20260429 19:59:33.589129 27838 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 19:59:33.593477 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 27838
I20260429 19:59:33.593559 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 19:59:33.593775 27719 external_mini_cluster.cc:1468] Setting key 1e9962452c20d0d6a7fd1ecee1c5564d
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:33.602351 27813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:33.590961 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:46721 (local address 127.27.17.254:37337)
0429 19:59:33.591126 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:33.591131 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:33.591886 (+ 755us) server_negotiation.cc:408] Connection header received
0429 19:59:33.592722 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:33.592725 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:33.592774 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:33.592852 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:33.594766 (+ 1914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.595420 (+ 654us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:33.596125 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.596316 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:33.598791 (+ 2475us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:33.598817 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:33.598820 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:33.598845 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:33.600455 (+ 1610us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:33.600955 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:33.600959 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:33.600961 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:33.601013 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:33.601312 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:33.601316 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:33.601318 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:33.601475 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:33.601581 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:33.602046 (+ 465us) server_negotiation.cc:300] Negotiation successful
0429 19:59:33.602187 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 19:59:33.603116 27969 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:33.591323 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37337 (local address 127.27.17.193:46721)
0429 19:59:33.591736 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:33.591770 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:33.592528 (+ 758us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:33.593000 (+ 472us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:33.593008 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:33.593410 (+ 402us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:33.594479 (+ 1069us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:33.594494 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.595559 (+ 1065us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:33.595563 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:33.595966 (+ 403us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:33.595973 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.596214 (+ 241us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:33.596869 (+ 655us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:33.596901 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:33.598628 (+ 1727us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:33.600588 (+ 1960us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:33.600593 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:33.600606 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:33.600849 (+ 243us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:33.601109 (+ 260us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:33.601112 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:33.601117 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:33.601216 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:33.601581 (+ 365us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:33.601588 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:33.601809 (+ 221us) client_negotiation.cc:770] Sending connection context
0429 19:59:33.602015 (+ 206us) client_negotiation.cc:241] Negotiation successful
0429 19:59:33.602250 (+ 235us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":97,"threads_started":1}
I20260429 19:59:33.604341 27967 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37337
I20260429 19:59:33.604602 27967 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:33.605114 27967 heartbeater.cc:507] Master 127.27.17.254:37337 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
I20260429 19:59:33.606663 27774 ts_manager.cc:194] Registered new tserver with Master: 3a439b7dd4804c41809dda7203ac4f00 (127.27.17.193:44341)
I20260429 19:59:33.608273 27774 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:46721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:33.646144 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37337
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:33.751959 27974 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:33.752177 27974 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:33.752238 27974 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:33.755532 27974 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:33.755592 27974 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:33.755681 27974 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 19:59:33.760396 27974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:33.761585 27974 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:33.762497 27974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:33.769534 27979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:33.769924 27980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:33.769963 27982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:33.770009 27974 server_base.cc:1061] running on GCE node
I20260429 19:59:33.770609 27974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:33.771440 27974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:33.772630 27974 hybrid_clock.cc:648] HybridClock initialized: now 1777492773772583 us; error 68 us; skew 500 ppm
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:33.775585 27974 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 19:59:33.776734 27974 webserver.cc:492] Webserver started at http://127.27.17.194:39587/ using document root <none> and password file <none>
I20260429 19:59:33.777325 27974 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:33.777376 27974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:33.777582 27974 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:33.779450 27974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "edd9c3a3ebdc424aadc2a115187b2d8d"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "2d279172e7f0b0a60cf80e8c8e4159da"
server_key_iv: "44f1c1f8d0028852b644a3e37d7f3bc5"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.779943 27974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "edd9c3a3ebdc424aadc2a115187b2d8d"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "2d279172e7f0b0a60cf80e8c8e4159da"
server_key_iv: "44f1c1f8d0028852b644a3e37d7f3bc5"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.783413 27974 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 19:59:33.785646 27989 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.786648 27974 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:33.786854 27974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "edd9c3a3ebdc424aadc2a115187b2d8d"
format_stamp: "Formatted at 2026-04-29 19:59:33 on dist-test-slave-01nr"
server_key: "2d279172e7f0b0a60cf80e8c8e4159da"
server_key_iv: "44f1c1f8d0028852b644a3e37d7f3bc5"
server_key_version: "encryptionkey@0"
I20260429 19:59:33.787110 27974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:33.811508 27974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:33.814462 27974 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:33.814682 27974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:33.815420 27974 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:33.816396 27974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:33.816475 27974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.816546 27974 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:33.816599 27974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:33.826567 27974 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:40189
I20260429 19:59:33.826607 28102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:40189 every 8 connection(s)
I20260429 19:59:33.827648 27974 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 19:59:33.832468 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 27974
I20260429 19:59:33.832584 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 19:59:33.832814 27719 external_mini_cluster.cc:1468] Setting key 070dbb58cdda9a8c26d224a6a46b73f0
Apr 29 19:59:33 dist-test-slave-01nr krb5kdc[27726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492773, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 19:59:33.842018 27813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:33.829663 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:37635 (local address 127.27.17.254:37337)
0429 19:59:33.829827 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:33.829832 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:33.830571 (+ 739us) server_negotiation.cc:408] Connection header received
0429 19:59:33.831492 (+ 921us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:33.831495 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:33.831546 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:33.831624 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:33.833425 (+ 1801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.834149 (+ 724us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:33.834862 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.835118 (+ 256us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:33.837962 (+ 2844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:33.837981 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:33.837984 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:33.838013 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:33.839849 (+ 1836us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:33.840421 (+ 572us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:33.840425 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:33.840427 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:33.840483 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:33.840792 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:33.840796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:33.840798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:33.841011 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:33.841120 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:33.841790 (+ 670us) server_negotiation.cc:300] Negotiation successful
0429 19:59:33.841894 (+ 104us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260429 19:59:33.842885 28105 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:33.829968 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37337 (local address 127.27.17.194:37635)
0429 19:59:33.830423 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:33.830458 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:33.831293 (+ 835us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:33.831823 (+ 530us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:33.831831 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:33.832262 (+ 431us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:33.833241 (+ 979us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:33.833256 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.834300 (+ 1044us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:33.834303 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:33.834718 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:33.834725 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:33.834993 (+ 268us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:33.835692 (+ 699us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:33.835726 (+ 34us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:33.837740 (+ 2014us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:33.840018 (+ 2278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:33.840025 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:33.840039 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:33.840297 (+ 258us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:33.840591 (+ 294us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:33.840594 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:33.840595 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:33.840695 (+ 100us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:33.841154 (+ 459us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:33.841162 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:33.841492 (+ 330us) client_negotiation.cc:770] Sending connection context
0429 19:59:33.841767 (+ 275us) client_negotiation.cc:241] Negotiation successful
0429 19:59:33.842029 (+ 262us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":93,"threads_started":1}
I20260429 19:59:33.844532 28103 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37337
I20260429 19:59:33.844810 28103 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 19:59:33.845535 28103 heartbeater.cc:507] Master 127.27.17.254:37337 requested a full tablet report, sending...
I20260429 19:59:33.846827 27774 ts_manager.cc:194] Registered new tserver with Master: edd9c3a3ebdc424aadc2a115187b2d8d (127.27.17.194:40189)
I20260429 19:59:33.847538 27774 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:37635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:33.891985 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37337
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:34.002942 28110 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:34.003199 28110 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:34.003260 28110 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:34.006656 28110 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:34.006733 28110 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:34.006817 28110 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 19:59:34.011523 28110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:34.012583 28110 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:34.013450 28110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:34.020334 28115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:34.020450 28110 server_base.cc:1061] running on GCE node
W20260429 19:59:34.020355 28118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:34.020360 28116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:34.020992 28110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:34.021521 28110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:34.022715 28110 hybrid_clock.cc:648] HybridClock initialized: now 1777492774022693 us; error 37 us; skew 500 ppm
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492774, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:34.025620 28110 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 19:59:34.026728 28110 webserver.cc:492] Webserver started at http://127.27.17.195:43665/ using document root <none> and password file <none>
I20260429 19:59:34.027521 28110 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:34.027618 28110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:34.027842 28110 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:34.029618 28110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "da5b410d475c4f428b5c337584203a19"
format_stamp: "Formatted at 2026-04-29 19:59:34 on dist-test-slave-01nr"
server_key: "2095cedeb78c183a4787593ed04edb8c"
server_key_iv: "4d179979e71edec51752ce1c4fac563b"
server_key_version: "encryptionkey@0"
I20260429 19:59:34.030126 28110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "da5b410d475c4f428b5c337584203a19"
format_stamp: "Formatted at 2026-04-29 19:59:34 on dist-test-slave-01nr"
server_key: "2095cedeb78c183a4787593ed04edb8c"
server_key_iv: "4d179979e71edec51752ce1c4fac563b"
server_key_version: "encryptionkey@0"
I20260429 19:59:34.033630 28110 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 19:59:34.036289 28125 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:34.037474 28110 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:34.037652 28110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "da5b410d475c4f428b5c337584203a19"
format_stamp: "Formatted at 2026-04-29 19:59:34 on dist-test-slave-01nr"
server_key: "2095cedeb78c183a4787593ed04edb8c"
server_key_iv: "4d179979e71edec51752ce1c4fac563b"
server_key_version: "encryptionkey@0"
I20260429 19:59:34.037776 28110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:34.051584 28110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:34.054454 28110 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:34.054687 28110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:34.055325 28110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:34.056245 28110 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:34.056325 28110 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:34.056398 28110 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:34.056452 28110 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:34.066103 28110 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:44517
I20260429 19:59:34.066120 28238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:44517 every 8 connection(s)
I20260429 19:59:34.067379 28110 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 19:59:34.068310 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28110
I20260429 19:59:34.068415 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 19:59:34.068681 27719 external_mini_cluster.cc:1468] Setting key 0abfe4f49da632106dad7314fa64f1a6
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[27726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492774, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:34.080610 27813 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:34.069650 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:52117 (local address 127.27.17.254:37337)
0429 19:59:34.070883 (+ 1233us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:34.070888 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:34.070905 (+ 17us) server_negotiation.cc:408] Connection header received
0429 19:59:34.071690 (+ 785us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:34.071693 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:34.071741 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:34.071810 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:34.073426 (+ 1616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.073940 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:34.074659 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.074807 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:34.077133 (+ 2326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:34.077154 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:34.077156 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:34.077182 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:34.078638 (+ 1456us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:34.079203 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:34.079206 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:34.079207 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:34.079252 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:34.079566 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:34.079569 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:34.079570 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:34.079723 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:34.079808 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:34.080364 (+ 556us) server_negotiation.cc:300] Negotiation successful
0429 19:59:34.080478 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1143}
I20260429 19:59:34.081192 28241 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:34.069991 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37337 (local address 127.27.17.195:52117)
0429 19:59:34.070507 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:34.070554 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:34.071450 (+ 896us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:34.071971 (+ 521us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:34.071979 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:34.072394 (+ 415us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:34.073258 (+ 864us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:34.073272 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.074096 (+ 824us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:34.074100 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:34.074525 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:34.074532 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.074713 (+ 181us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:34.075409 (+ 696us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:34.075429 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:34.076930 (+ 1501us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:34.078757 (+ 1827us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:34.078763 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:34.078774 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:34.079114 (+ 340us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:34.079339 (+ 225us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:34.079341 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:34.079343 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:34.079437 (+ 94us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:34.079829 (+ 392us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:34.079834 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:34.080085 (+ 251us) client_negotiation.cc:770] Sending connection context
0429 19:59:34.080259 (+ 174us) client_negotiation.cc:241] Negotiation successful
0429 19:59:34.080461 (+ 202us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":104,"threads_started":1}
I20260429 19:59:34.082537 28239 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37337
I20260429 19:59:34.082806 28239 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:34.083616 28239 heartbeater.cc:507] Master 127.27.17.254:37337 requested a full tablet report, sending...
I20260429 19:59:34.084667 27774 ts_manager.cc:194] Registered new tserver with Master: da5b410d475c4f428b5c337584203a19 (127.27.17.195:44517)
I20260429 19:59:34.085254 27774 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:52117
I20260429 19:59:34.095587 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[27726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 19:59:34.120497 28246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:34.111224 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.27.17.193:44341)
0429 19:59:34.111516 (+ 292us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:34.111522 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:34.111545 (+ 23us) server_negotiation.cc:408] Connection header received
0429 19:59:34.111605 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:34.111609 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:34.111786 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:34.111901 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:34.112708 (+ 807us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.113439 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:34.114121 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.114329 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:34.116578 (+ 2249us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:34.116607 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:34.116619 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:34.116648 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:34.118515 (+ 1867us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:34.119106 (+ 591us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:34.119111 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:34.119113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:34.119171 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:34.119485 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:34.119488 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:34.119490 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:34.119689 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:34.119868 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:34.120098 (+ 230us) server_negotiation.cc:300] Negotiation successful
0429 19:59:34.120277 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":94,"threads_started":1}
W20260429 19:59:34.121490 27879 server_base.cc:1143] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40430
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[27726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[27726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 19:59:34.146940 28246 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:34.138435 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.27.17.193:44341)
0429 19:59:34.138645 (+ 210us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:34.138698 (+ 53us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:34.138722 (+ 24us) server_negotiation.cc:408] Connection header received
0429 19:59:34.138892 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:34.138958 (+ 66us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:34.139089 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:34.139174 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:34.140101 (+ 927us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.140649 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:34.141338 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:34.141506 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:34.143883 (+ 2377us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:34.143905 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:34.143908 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:34.143935 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:34.145238 (+ 1303us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:34.145595 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:34.145598 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:34.145599 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:34.145641 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:34.145857 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:34.145863 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:34.145866 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:34.146017 (+ 151us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:34.146150 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:34.146544 (+ 394us) server_negotiation.cc:300] Negotiation successful
0429 19:59:34.146689 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260429 19:59:34.148212 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 27838
I20260429 19:59:34.154661 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 27974
I20260429 19:59:34.163175 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28110
I20260429 19:59:34.169271 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 27742
2026-04-29T19:59:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3236 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[28253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[28253](info): set up 2 sockets
Apr 29 19:59:34 dist-test-slave-01nr krb5kdc[28253](info): commencing operation
krb5kdc: starting...
W20260429 19:59:36.212399 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 19:59:36 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T19:59:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T19:59:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:36.358485 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34715
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:34715
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:36.460948 28269 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:36.461253 28269 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:36.461360 28269 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:36.464892 28269 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 19:59:36.464991 28269 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:36.465037 28269 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:36.465101 28269 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 19:59:36.465145 28269 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 19:59:36.469555 28269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:34715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:36.470639 28269 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:36.471580 28269 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:36.477272 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:36.477273 28274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:36.477272 28275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:36.477595 28269 server_base.cc:1061] running on GCE node
I20260429 19:59:36.478178 28269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:36.479214 28269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:36.480397 28269 hybrid_clock.cc:648] HybridClock initialized: now 1777492776480377 us; error 43 us; skew 500 ppm
Apr 29 19:59:36 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492776, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:36.483163 28269 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 19:59:36.484212 28269 webserver.cc:492] Webserver started at http://127.27.17.254:40629/ using document root <none> and password file <none>
I20260429 19:59:36.484759 28269 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:36.484807 28269 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:36.485014 28269 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:36.486701 28269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "cdb866c72b804f508302a2b6eb1e1986"
format_stamp: "Formatted at 2026-04-29 19:59:36 on dist-test-slave-01nr"
server_key: "7791f0b1b9c533d2da89e1049bcc2b14"
server_key_iv: "f7c072d8078bea65f4b3a2ba3f9434f1"
server_key_version: "encryptionkey@0"
I20260429 19:59:36.487253 28269 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "cdb866c72b804f508302a2b6eb1e1986"
format_stamp: "Formatted at 2026-04-29 19:59:36 on dist-test-slave-01nr"
server_key: "7791f0b1b9c533d2da89e1049bcc2b14"
server_key_iv: "f7c072d8078bea65f4b3a2ba3f9434f1"
server_key_version: "encryptionkey@0"
I20260429 19:59:36.490486 28269 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260429 19:59:36.492720 28284 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:36.493723 28269 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:36.493845 28269 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "cdb866c72b804f508302a2b6eb1e1986"
format_stamp: "Formatted at 2026-04-29 19:59:36 on dist-test-slave-01nr"
server_key: "7791f0b1b9c533d2da89e1049bcc2b14"
server_key_iv: "f7c072d8078bea65f4b3a2ba3f9434f1"
server_key_version: "encryptionkey@0"
I20260429 19:59:36.493949 28269 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:36.509485 28269 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:36.517390 28269 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:36.517616 28269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:36.525318 28336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:34715 every 8 connection(s)
I20260429 19:59:36.525315 28269 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:34715
I20260429 19:59:36.526399 28269 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 19:59:36.529235 28337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:36.534289 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28269
I20260429 19:59:36.534427 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 19:59:36.534588 28337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986: Bootstrap starting.
I20260429 19:59:36.534700 27719 external_mini_cluster.cc:1468] Setting key 5dbbda9b93ef19f8f0a3cb2eb1e6013e
I20260429 19:59:36.537173 28337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:36.538051 28337 log.cc:826] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:36.540118 28337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986: No bootstrap required, opened a new log
Apr 29 19:59:36 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:36.543351 28337 raft_consensus.cc:359] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb866c72b804f508302a2b6eb1e1986" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34715 } }
I20260429 19:59:36.543534 28337 raft_consensus.cc:385] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:36.543612 28337 raft_consensus.cc:740] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb866c72b804f508302a2b6eb1e1986, State: Initialized, Role: FOLLOWER
I20260429 19:59:36.544039 28337 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb866c72b804f508302a2b6eb1e1986" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34715 } }
I20260429 19:59:36.544159 28337 raft_consensus.cc:399] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 19:59:36.544271 28337 raft_consensus.cc:493] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 19:59:36.544380 28337 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:36.545506 28337 raft_consensus.cc:515] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb866c72b804f508302a2b6eb1e1986" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34715 } }
I20260429 19:59:36.545913 28337 leader_election.cc:304] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdb866c72b804f508302a2b6eb1e1986; no voters:
I20260429 19:59:36.546270 28337 leader_election.cc:290] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 19:59:36.546348 28342 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:36.546515 28342 raft_consensus.cc:697] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [term 1 LEADER]: Becoming Leader. State: Replica: cdb866c72b804f508302a2b6eb1e1986, State: Running, Role: LEADER
I20260429 19:59:36.546795 28342 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb866c72b804f508302a2b6eb1e1986" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34715 } }
I20260429 19:59:36.547495 28337 sys_catalog.cc:565] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 19:59:36.548440 28344 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdb866c72b804f508302a2b6eb1e1986. Latest consensus state: current_term: 1 leader_uuid: "cdb866c72b804f508302a2b6eb1e1986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb866c72b804f508302a2b6eb1e1986" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34715 } } }
I20260429 19:59:36.548475 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdb866c72b804f508302a2b6eb1e1986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdb866c72b804f508302a2b6eb1e1986" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34715 } } }
I20260429 19:59:36.548568 28344 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:36.548588 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:36.549288 28340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:36.536054 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.27.17.254:34715)
0429 19:59:36.536496 (+ 442us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:36.536506 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:36.536540 (+ 34us) server_negotiation.cc:408] Connection header received
0429 19:59:36.537082 (+ 542us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:36.537099 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:36.537458 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:36.537732 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:36.538764 (+ 1032us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:36.539608 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:36.540454 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:36.540645 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:36.543204 (+ 2559us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:36.543225 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:36.543236 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:36.543266 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:36.545419 (+ 2153us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:36.545890 (+ 471us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:36.545897 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:36.545903 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:36.545999 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:36.546442 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:36.546446 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:36.546448 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:36.546943 (+ 495us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:36.547708 (+ 765us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:36.547940 (+ 232us) server_negotiation.cc:300] Negotiation successful
0429 19:59:36.548220 (+ 280us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":166,"threads_started":1}
W20260429 19:59:36.552522 28357 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 19:59:36.552611 28357 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 19:59:36.552695 28350 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 19:59:36.553625 28350 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 19:59:36.558951 28350 catalog_manager.cc:1357] Generated new cluster ID: 433523a0064849bf9aca8cc52d016dce
I20260429 19:59:36.559034 28350 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 19:59:36.578177 28350 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 19:59:36.579169 28350 catalog_manager.cc:1514] Loading token signing keys...
I20260429 19:59:36.590167 28350 catalog_manager.cc:6044] T 00000000000000000000000000000000 P cdb866c72b804f508302a2b6eb1e1986: Generated new TSK 0
I20260429 19:59:36.590991 28350 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:36.655851 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34715
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:36.763691 28365 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:36.763927 28365 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:36.763988 28365 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:36.767364 28365 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:36.767431 28365 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:36.767514 28365 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 19:59:36.771924 28365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34715
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:36.773077 28365 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:36.773936 28365 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:36.781085 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:36.781158 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:36.781287 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:36.781538 28365 server_base.cc:1061] running on GCE node
I20260429 19:59:36.782008 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:36.782760 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:36.783941 28365 hybrid_clock.cc:648] HybridClock initialized: now 1777492776783906 us; error 54 us; skew 500 ppm
Apr 29 19:59:36 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492776, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:36.787029 28365 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 19:59:36.788277 28365 webserver.cc:492] Webserver started at http://127.27.17.193:35611/ using document root <none> and password file <none>
I20260429 19:59:36.788879 28365 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:36.788933 28365 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:36.789167 28365 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:36.791087 28365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "0c466c2de7d24cd7b5914378da4e7038"
format_stamp: "Formatted at 2026-04-29 19:59:36 on dist-test-slave-01nr"
server_key: "af98a1f30e0e7d04417010e10de1f38d"
server_key_iv: "79e4c92b76622424ee6e4cd0fac4c864"
server_key_version: "encryptionkey@0"
I20260429 19:59:36.791627 28365 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "0c466c2de7d24cd7b5914378da4e7038"
format_stamp: "Formatted at 2026-04-29 19:59:36 on dist-test-slave-01nr"
server_key: "af98a1f30e0e7d04417010e10de1f38d"
server_key_iv: "79e4c92b76622424ee6e4cd0fac4c864"
server_key_version: "encryptionkey@0"
I20260429 19:59:36.795320 28365 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 19:59:36.797693 28380 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:36.798972 28365 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 19:59:36.799122 28365 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "0c466c2de7d24cd7b5914378da4e7038"
format_stamp: "Formatted at 2026-04-29 19:59:36 on dist-test-slave-01nr"
server_key: "af98a1f30e0e7d04417010e10de1f38d"
server_key_iv: "79e4c92b76622424ee6e4cd0fac4c864"
server_key_version: "encryptionkey@0"
I20260429 19:59:36.799266 28365 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:36.811064 28365 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:36.814036 28365 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:36.814250 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:36.814848 28365 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:36.815837 28365 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:36.815896 28365 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:36.815969 28365 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:36.816004 28365 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:36.827492 28365 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:35195
I20260429 19:59:36.827567 28493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:35195 every 8 connection(s)
I20260429 19:59:36.828601 28365 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 19:59:36.832381 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28365
I20260429 19:59:36.832523 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 19:59:36.832818 27719 external_mini_cluster.cc:1468] Setting key 85b28bd92424572e6b5a3acb27cbd9a7
Apr 29 19:59:36 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492776, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 19:59:36.843279 28340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:36.830537 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:60639 (local address 127.27.17.254:34715)
0429 19:59:36.830721 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:36.830724 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:36.831457 (+ 733us) server_negotiation.cc:408] Connection header received
0429 19:59:36.832399 (+ 942us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:36.832403 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:36.832464 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:36.832547 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:36.834825 (+ 2278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:36.835446 (+ 621us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:36.836429 (+ 983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:36.836670 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:36.839194 (+ 2524us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:36.839228 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:36.839233 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:36.839277 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:36.841033 (+ 1756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:36.841612 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:36.841616 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:36.841619 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:36.841686 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:36.842009 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:36.842015 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:36.842019 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:36.842263 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:36.842391 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:36.842986 (+ 595us) server_negotiation.cc:300] Negotiation successful
0429 19:59:36.843109 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260429 19:59:36.844070 28496 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:36.830828 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34715 (local address 127.27.17.193:60639)
0429 19:59:36.831313 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:36.831348 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:36.832191 (+ 843us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:36.832850 (+ 659us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:36.832865 (+ 15us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:36.833457 (+ 592us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:36.834634 (+ 1177us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:36.834653 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:36.835603 (+ 950us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:36.835611 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:36.836264 (+ 653us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:36.836277 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:36.836531 (+ 254us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:36.837162 (+ 631us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:36.837184 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:36.838989 (+ 1805us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:36.841199 (+ 2210us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:36.841206 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:36.841217 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:36.841493 (+ 276us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:36.841788 (+ 295us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:36.841791 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:36.841792 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:36.841902 (+ 110us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:36.842395 (+ 493us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:36.842401 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:36.842630 (+ 229us) client_negotiation.cc:770] Sending connection context
0429 19:59:36.842870 (+ 240us) client_negotiation.cc:241] Negotiation successful
0429 19:59:36.843167 (+ 297us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":107,"threads_started":1}
I20260429 19:59:36.845359 28494 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34715
I20260429 19:59:36.845737 28494 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
I20260429 19:59:36.846341 28494 heartbeater.cc:507] Master 127.27.17.254:34715 requested a full tablet report, sending...
I20260429 19:59:36.848104 28301 ts_manager.cc:194] Registered new tserver with Master: 0c466c2de7d24cd7b5914378da4e7038 (127.27.17.193:35195)
I20260429 19:59:36.849900 28301 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:60639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:36.894773 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34715
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:37.006042 28501 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:37.006317 28501 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:37.006433 28501 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:37.010077 28501 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:37.010190 28501 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:37.010303 28501 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 19:59:37.014868 28501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34715
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:37.016127 28501 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:37.017033 28501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:37.024077 28507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:37.024155 28506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:37.024276 28509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:37.024317 28501 server_base.cc:1061] running on GCE node
I20260429 19:59:37.024804 28501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:37.025398 28501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:37.026582 28501 hybrid_clock.cc:648] HybridClock initialized: now 1777492777026568 us; error 33 us; skew 500 ppm
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:37.029587 28501 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 19:59:37.031185 28501 webserver.cc:492] Webserver started at http://127.27.17.194:35361/ using document root <none> and password file <none>
I20260429 19:59:37.031819 28501 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:37.031910 28501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:37.032161 28501 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:37.034034 28501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "11b1cbdb69b746bda74be3972ca48823"
format_stamp: "Formatted at 2026-04-29 19:59:37 on dist-test-slave-01nr"
server_key: "b132665a067f3d3860f1f00256a164bf"
server_key_iv: "94d5c9ec181c259f8bff6e8e6446ed23"
server_key_version: "encryptionkey@0"
I20260429 19:59:37.034565 28501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "11b1cbdb69b746bda74be3972ca48823"
format_stamp: "Formatted at 2026-04-29 19:59:37 on dist-test-slave-01nr"
server_key: "b132665a067f3d3860f1f00256a164bf"
server_key_iv: "94d5c9ec181c259f8bff6e8e6446ed23"
server_key_version: "encryptionkey@0"
I20260429 19:59:37.038084 28501 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260429 19:59:37.040330 28516 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:37.041312 28501 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:37.041448 28501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "11b1cbdb69b746bda74be3972ca48823"
format_stamp: "Formatted at 2026-04-29 19:59:37 on dist-test-slave-01nr"
server_key: "b132665a067f3d3860f1f00256a164bf"
server_key_iv: "94d5c9ec181c259f8bff6e8e6446ed23"
server_key_version: "encryptionkey@0"
I20260429 19:59:37.041556 28501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:37.055596 28501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:37.058588 28501 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:37.058842 28501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:37.059491 28501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:37.060436 28501 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:37.060506 28501 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:37.060577 28501 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:37.060631 28501 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:37.069677 28501 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:41991
I20260429 19:59:37.069768 28629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:41991 every 8 connection(s)
I20260429 19:59:37.070711 28501 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 19:59:37.071147 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28501
I20260429 19:59:37.071264 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 19:59:37.071549 27719 external_mini_cluster.cc:1468] Setting key 9b184c702c5517124adbda287c8b4e95
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 19:59:37.084874 28340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.072529 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:58707 (local address 127.27.17.254:34715)
0429 19:59:37.072713 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.072719 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.073589 (+ 870us) server_negotiation.cc:408] Connection header received
0429 19:59:37.074533 (+ 944us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.074539 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.074609 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.074698 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:37.076927 (+ 2229us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.077455 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.078472 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.078638 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.081275 (+ 2637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:37.081301 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:37.081304 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:37.081331 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:37.082794 (+ 1463us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.083363 (+ 569us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.083368 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.083372 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.083422 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.083741 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.083744 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.083746 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.083906 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:37.084016 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.084571 (+ 555us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.084700 (+ 129us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260429 19:59:37.085537 28632 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.072956 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34715 (local address 127.27.17.194:58707)
0429 19:59:37.073431 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.073474 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.074294 (+ 820us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.074936 (+ 642us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.074950 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.075546 (+ 596us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:37.076745 (+ 1199us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.076768 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.077603 (+ 835us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.077608 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.078312 (+ 704us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.078324 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.078608 (+ 284us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.079318 (+ 710us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:37.079349 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:37.081106 (+ 1757us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:37.082977 (+ 1871us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:37.082984 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:37.083000 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:37.083249 (+ 249us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:37.083533 (+ 284us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:37.083536 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:37.083538 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:37.083637 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:37.084016 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:37.084023 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:37.084309 (+ 286us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.084519 (+ 210us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.084770 (+ 251us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260429 19:59:37.086984 28630 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34715
I20260429 19:59:37.087430 28630 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:37.088053 28630 heartbeater.cc:507] Master 127.27.17.254:34715 requested a full tablet report, sending...
I20260429 19:59:37.089314 28301 ts_manager.cc:194] Registered new tserver with Master: 11b1cbdb69b746bda74be3972ca48823 (127.27.17.194:41991)
I20260429 19:59:37.090058 28301 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:58707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:37.130832 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34715
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:37.238014 28637 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:37.238250 28637 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:37.238312 28637 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:37.241592 28637 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:37.241657 28637 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:37.241737 28637 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 19:59:37.245941 28637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34715
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:37.247134 28637 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:37.247931 28637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:37.254412 28643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:37.254417 28645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:37.254508 28642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:37.254781 28637 server_base.cc:1061] running on GCE node
I20260429 19:59:37.255187 28637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:37.255752 28637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:37.256930 28637 hybrid_clock.cc:648] HybridClock initialized: now 1777492777256910 us; error 38 us; skew 500 ppm
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:37.259842 28637 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 19:59:37.260977 28637 webserver.cc:492] Webserver started at http://127.27.17.195:37041/ using document root <none> and password file <none>
I20260429 19:59:37.261603 28637 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:37.261677 28637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:37.261897 28637 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:37.263707 28637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "6fb911cad36947cdadd02455e5534a6f"
format_stamp: "Formatted at 2026-04-29 19:59:37 on dist-test-slave-01nr"
server_key: "8c81219f6e7712cc57eb3b2f146f4108"
server_key_iv: "87d695e1e04234bacf28bf87a04c9c15"
server_key_version: "encryptionkey@0"
I20260429 19:59:37.264225 28637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "6fb911cad36947cdadd02455e5534a6f"
format_stamp: "Formatted at 2026-04-29 19:59:37 on dist-test-slave-01nr"
server_key: "8c81219f6e7712cc57eb3b2f146f4108"
server_key_iv: "87d695e1e04234bacf28bf87a04c9c15"
server_key_version: "encryptionkey@0"
I20260429 19:59:37.267817 28637 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260429 19:59:37.270264 28652 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:37.271385 28637 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 19:59:37.271534 28637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "6fb911cad36947cdadd02455e5534a6f"
format_stamp: "Formatted at 2026-04-29 19:59:37 on dist-test-slave-01nr"
server_key: "8c81219f6e7712cc57eb3b2f146f4108"
server_key_iv: "87d695e1e04234bacf28bf87a04c9c15"
server_key_version: "encryptionkey@0"
I20260429 19:59:37.271648 28637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:37.284121 28637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:37.287250 28637 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:37.287464 28637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:37.288064 28637 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:37.289003 28637 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:37.289077 28637 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:37.289150 28637 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:37.289196 28637 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:37.299141 28637 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:41797
I20260429 19:59:37.299165 28765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:41797 every 8 connection(s)
I20260429 19:59:37.300091 28637 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 19:59:37.306881 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28637
I20260429 19:59:37.307049 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 19:59:37.307354 27719 external_mini_cluster.cc:1468] Setting key a6ab0bb5445d38e67dc111053e456b22
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:37.313457 28340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.301868 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:44257 (local address 127.27.17.254:34715)
0429 19:59:37.302044 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.302047 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.302656 (+ 609us) server_negotiation.cc:408] Connection header received
0429 19:59:37.303543 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.303546 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.303608 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.303684 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:37.305794 (+ 2110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.306304 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.307178 (+ 874us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.307550 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.309922 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:37.309945 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:37.309948 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:37.309976 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:37.311500 (+ 1524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.312026 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.312034 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.312036 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.312082 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.312398 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.312401 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.312403 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.312555 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:37.312656 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.313160 (+ 504us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.313281 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260429 19:59:37.314118 28768 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.302116 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34715 (local address 127.27.17.195:44257)
0429 19:59:37.302508 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.302541 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.303353 (+ 812us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.303905 (+ 552us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.303913 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.304296 (+ 383us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:37.305630 (+ 1334us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.305648 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.306423 (+ 775us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.306426 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.306981 (+ 555us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.306991 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.307253 (+ 262us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.307942 (+ 689us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:37.307962 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:37.309759 (+ 1797us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:37.311631 (+ 1872us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:37.311637 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:37.311651 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:37.311910 (+ 259us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:37.312188 (+ 278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:37.312191 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:37.312193 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:37.312290 (+ 97us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:37.312669 (+ 379us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:37.312676 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:37.312914 (+ 238us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.313118 (+ 204us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.313389 (+ 271us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":99,"threads_started":1}
I20260429 19:59:37.315285 28766 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34715
I20260429 19:59:37.315549 28766 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:37.316102 28766 heartbeater.cc:507] Master 127.27.17.254:34715 requested a full tablet report, sending...
I20260429 19:59:37.317193 28301 ts_manager.cc:194] Registered new tserver with Master: 6fb911cad36947cdadd02455e5534a6f (127.27.17.195:41797)
I20260429 19:59:37.317776 28301 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:44257
I20260429 19:59:37.321205 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 19:59:37.331176 28340 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.324121 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60994 (local address 127.27.17.254:34715)
0429 19:59:37.324248 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.324252 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.324395 (+ 143us) server_negotiation.cc:408] Connection header received
0429 19:59:37.324498 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.324501 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.324543 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.324611 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:37.325434 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.325922 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.326533 (+ 611us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.326661 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.327962 (+ 1301us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:37.327980 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:37.327982 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:37.328013 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:37.329568 (+ 1555us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.330001 (+ 433us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.330005 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.330009 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.330063 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.330309 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.330313 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.330315 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.330491 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:37.330633 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.330863 (+ 230us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.331004 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 19:59:37.335722 28301 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 19:59:37.338094 28301 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:37.351577 28777 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.347463 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:41991 (local address 127.0.0.1:55620)
0429 19:59:37.348147 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.348162 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.348242 (+ 80us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.348713 (+ 471us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.348720 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.348748 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:37.349091 (+ 343us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.349096 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.350149 (+ 1053us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.350153 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.351030 (+ 877us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.351050 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.351192 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.351251 (+ 59us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.351346 (+ 95us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.351422 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":125,"threads_started":1}
I20260429 19:59:37.352085 28781 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.347369 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:35195 (local address 127.0.0.1:36008)
0429 19:59:37.347757 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.347769 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.347914 (+ 145us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.348839 (+ 925us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.348842 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.348856 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:37.349089 (+ 233us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.349096 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.350694 (+ 1598us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.350698 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.351691 (+ 993us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.351700 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.351791 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.351806 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.351851 (+ 45us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.351906 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"spinlock_wait_cycles":4864,"thread_start_us":106,"threads_started":1}
I20260429 19:59:37.352159 28779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.347167 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:55620 (local address 127.27.17.194:41991)
0429 19:59:37.348191 (+ 1024us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.348199 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.348221 (+ 22us) server_negotiation.cc:408] Connection header received
0429 19:59:37.348372 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.348376 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.348540 (+ 164us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.348693 (+ 153us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:37.349213 (+ 520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.350018 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.351190 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.351703 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.351841 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.351908 (+ 67us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.351997 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":116,"threads_started":1}
I20260429 19:59:37.352085 28780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.346846 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:41797 (local address 127.0.0.1:45288)
0429 19:59:37.347716 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.347756 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.347914 (+ 158us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.348869 (+ 955us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.348872 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.348890 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:37.349180 (+ 290us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.349191 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.350699 (+ 1508us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.350702 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.351691 (+ 989us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.351700 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.351793 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.351806 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.351867 (+ 61us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.351906 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"spinlock_wait_cycles":9216,"thread_start_us":135,"threads_started":1}
I20260429 19:59:37.353101 28778 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.346974 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45288 (local address 127.27.17.195:41797)
0429 19:59:37.348193 (+ 1219us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.348199 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.348221 (+ 22us) server_negotiation.cc:408] Connection header received
0429 19:59:37.348346 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.348355 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.348540 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.348693 (+ 153us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:37.349356 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.350582 (+ 1226us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.352163 (+ 1581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.352685 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.352782 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.352858 (+ 76us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.352940 (+ 82us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":149,"threads_started":1}
I20260429 19:59:37.354233 28782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.347764 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36008 (local address 127.27.17.193:35195)
0429 19:59:37.348438 (+ 674us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.348443 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.348459 (+ 16us) server_negotiation.cc:408] Connection header received
0429 19:59:37.348524 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.348527 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.348680 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.349239 (+ 559us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:37.349399 (+ 160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.350549 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.353297 (+ 2748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.353830 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.353927 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.354013 (+ 86us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.354111 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":71,"threads_started":1}
I20260429 19:59:37.355609 28564 tablet_service.cc:1511] Processing CreateTablet for tablet 401c688fab2647abbff21ca62886fc1f (DEFAULT_TABLE table=test-table [id=4664a85a488b404099875f5e46054cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:37.355829 28700 tablet_service.cc:1511] Processing CreateTablet for tablet 401c688fab2647abbff21ca62886fc1f (DEFAULT_TABLE table=test-table [id=4664a85a488b404099875f5e46054cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:37.356535 28564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401c688fab2647abbff21ca62886fc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:37.356720 28700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401c688fab2647abbff21ca62886fc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:37.356897 28428 tablet_service.cc:1511] Processing CreateTablet for tablet 401c688fab2647abbff21ca62886fc1f (DEFAULT_TABLE table=test-table [id=4664a85a488b404099875f5e46054cb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:37.357736 28428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 401c688fab2647abbff21ca62886fc1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:37.361864 28783 tablet_bootstrap.cc:492] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: Bootstrap starting.
I20260429 19:59:37.363260 28784 tablet_bootstrap.cc:492] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: Bootstrap starting.
I20260429 19:59:37.363569 28785 tablet_bootstrap.cc:492] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: Bootstrap starting.
I20260429 19:59:37.364699 28783 tablet_bootstrap.cc:654] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:37.365209 28784 tablet_bootstrap.cc:654] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:37.365345 28785 tablet_bootstrap.cc:654] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:37.365561 28783 log.cc:826] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:37.365958 28784 log.cc:826] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:37.366056 28785 log.cc:826] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:37.367592 28783 tablet_bootstrap.cc:492] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: No bootstrap required, opened a new log
I20260429 19:59:37.367767 28784 tablet_bootstrap.cc:492] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: No bootstrap required, opened a new log
I20260429 19:59:37.367770 28785 tablet_bootstrap.cc:492] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: No bootstrap required, opened a new log
I20260429 19:59:37.367797 28783 ts_tablet_manager.cc:1403] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260429 19:59:37.367947 28784 ts_tablet_manager.cc:1403] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260429 19:59:37.367956 28785 ts_tablet_manager.cc:1403] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260429 19:59:37.371168 28785 raft_consensus.cc:359] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.371464 28785 raft_consensus.cc:385] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:37.371546 28785 raft_consensus.cc:740] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c466c2de7d24cd7b5914378da4e7038, State: Initialized, Role: FOLLOWER
I20260429 19:59:37.371646 28783 raft_consensus.cc:359] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.371646 28784 raft_consensus.cc:359] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.371853 28784 raft_consensus.cc:385] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:37.371879 28783 raft_consensus.cc:385] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:37.371942 28784 raft_consensus.cc:740] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fb911cad36947cdadd02455e5534a6f, State: Initialized, Role: FOLLOWER
I20260429 19:59:37.371956 28783 raft_consensus.cc:740] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b1cbdb69b746bda74be3972ca48823, State: Initialized, Role: FOLLOWER
I20260429 19:59:37.372112 28785 consensus_queue.cc:260] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.372387 28784 consensus_queue.cc:260] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.372387 28783 consensus_queue.cc:260] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.373224 28784 ts_tablet_manager.cc:1434] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260429 19:59:37.373224 28783 ts_tablet_manager.cc:1434] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260429 19:59:37.373221 28785 ts_tablet_manager.cc:1434] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 19:59:37.373584 28494 heartbeater.cc:499] Master 127.27.17.254:34715 was elected leader, sending a full tablet report...
I20260429 19:59:37.373582 28630 heartbeater.cc:499] Master 127.27.17.254:34715 was elected leader, sending a full tablet report...
I20260429 19:59:37.374611 28766 heartbeater.cc:499] Master 127.27.17.254:34715 was elected leader, sending a full tablet report...
I20260429 19:59:37.528820 28791 raft_consensus.cc:493] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:37.529022 28791 raft_consensus.cc:515] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.530280 28791 leader_election.cc:290] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6fb911cad36947cdadd02455e5534a6f (127.27.17.195:41797), 0c466c2de7d24cd7b5914378da4e7038 (127.27.17.193:35195)
I20260429 19:59:37.533751 28632 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.530520 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:41797 (local address 127.27.17.194:51515)
0429 19:59:37.530648 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.530663 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.530780 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.531251 (+ 471us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.531254 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.531274 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:37.531539 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.531549 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.532441 (+ 892us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.532445 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.533352 (+ 907us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.533360 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.533491 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.533508 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.533556 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.533623 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260429 19:59:37.534271 28778 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.530839 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:51515 (local address 127.27.17.195:41797)
0429 19:59:37.531005 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.531010 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.531024 (+ 14us) server_negotiation.cc:408] Connection header received
0429 19:59:37.531092 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.531096 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.531139 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.531234 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:37.531673 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.532328 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.533479 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.533974 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.534015 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.534080 (+ 65us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.534137 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260429 19:59:37.534420 28792 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.530520 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:35195 (local address 127.27.17.194:45453)
0429 19:59:37.531168 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:37.531182 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:37.531333 (+ 151us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:37.531612 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:37.531615 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:37.531632 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:37.531931 (+ 299us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.531938 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.532800 (+ 862us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.532817 (+ 17us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:37.533395 (+ 578us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:37.533401 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.534124 (+ 723us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.534138 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 19:59:37.534218 (+ 80us) client_negotiation.cc:241] Negotiation successful
0429 19:59:37.534265 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":266,"thread_start_us":149,"threads_started":1}
I20260429 19:59:37.534448 28782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.530625 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:45453 (local address 127.27.17.193:35195)
0429 19:59:37.530771 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.530775 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.531264 (+ 489us) server_negotiation.cc:408] Connection header received
0429 19:59:37.531452 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.531456 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.531504 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.531586 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:37.532080 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.532682 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.533554 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.534036 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.534067 (+ 31us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.534267 (+ 200us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.534335 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 19:59:37.534925 28720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401c688fab2647abbff21ca62886fc1f" candidate_uuid: "11b1cbdb69b746bda74be3972ca48823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb911cad36947cdadd02455e5534a6f" is_pre_election: true
I20260429 19:59:37.535220 28720 raft_consensus.cc:2468] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b1cbdb69b746bda74be3972ca48823 in term 0.
I20260429 19:59:37.535477 28448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401c688fab2647abbff21ca62886fc1f" candidate_uuid: "11b1cbdb69b746bda74be3972ca48823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c466c2de7d24cd7b5914378da4e7038" is_pre_election: true
I20260429 19:59:37.535707 28448 raft_consensus.cc:2468] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11b1cbdb69b746bda74be3972ca48823 in term 0.
I20260429 19:59:37.535718 28519 leader_election.cc:304] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11b1cbdb69b746bda74be3972ca48823, 6fb911cad36947cdadd02455e5534a6f; no voters:
I20260429 19:59:37.535998 28791 raft_consensus.cc:2804] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:37.536072 28791 raft_consensus.cc:493] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:37.536121 28791 raft_consensus.cc:3060] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:37.537058 28791 raft_consensus.cc:515] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.537482 28791 leader_election.cc:290] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [CANDIDATE]: Term 1 election: Requested vote from peers 6fb911cad36947cdadd02455e5534a6f (127.27.17.195:41797), 0c466c2de7d24cd7b5914378da4e7038 (127.27.17.193:35195)
I20260429 19:59:37.537861 28720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401c688fab2647abbff21ca62886fc1f" candidate_uuid: "11b1cbdb69b746bda74be3972ca48823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fb911cad36947cdadd02455e5534a6f"
I20260429 19:59:37.537874 28448 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "401c688fab2647abbff21ca62886fc1f" candidate_uuid: "11b1cbdb69b746bda74be3972ca48823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c466c2de7d24cd7b5914378da4e7038"
I20260429 19:59:37.537990 28720 raft_consensus.cc:3060] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:37.537990 28448 raft_consensus.cc:3060] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:37.539233 28720 raft_consensus.cc:2468] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b1cbdb69b746bda74be3972ca48823 in term 1.
I20260429 19:59:37.539233 28448 raft_consensus.cc:2468] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11b1cbdb69b746bda74be3972ca48823 in term 1.
I20260429 19:59:37.539546 28517 leader_election.cc:304] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c466c2de7d24cd7b5914378da4e7038, 11b1cbdb69b746bda74be3972ca48823; no voters:
I20260429 19:59:37.539727 28791 raft_consensus.cc:2804] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:37.540019 28791 raft_consensus.cc:697] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [term 1 LEADER]: Becoming Leader. State: Replica: 11b1cbdb69b746bda74be3972ca48823, State: Running, Role: LEADER
I20260429 19:59:37.540367 28791 consensus_queue.cc:237] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } }
I20260429 19:59:37.543460 28301 catalog_manager.cc:5671] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11b1cbdb69b746bda74be3972ca48823 (127.27.17.194). New cstate: current_term: 1 leader_uuid: "11b1cbdb69b746bda74be3972ca48823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fb911cad36947cdadd02455e5534a6f" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11b1cbdb69b746bda74be3972ca48823" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 41991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c466c2de7d24cd7b5914378da4e7038" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35195 } health_report { overall_health: UNKNOWN } } }
W20260429 19:59:37.551025 28767 tablet.cc:2404] T 401c688fab2647abbff21ca62886fc1f P 6fb911cad36947cdadd02455e5534a6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 19:59:37.572160 28631 tablet.cc:2404] T 401c688fab2647abbff21ca62886fc1f P 11b1cbdb69b746bda74be3972ca48823: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 19:59:37.579970 28495 tablet.cc:2404] T 401c688fab2647abbff21ca62886fc1f P 0c466c2de7d24cd7b5914378da4e7038: Can't schedule compaction. Clean time has not been advanced past its initial value.
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 19:59:37.591879 28782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.582714 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36024 (local address 127.27.17.193:35195)
0429 19:59:37.582871 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.582875 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.583140 (+ 265us) server_negotiation.cc:408] Connection header received
0429 19:59:37.583222 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.583224 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.583273 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.583357 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:37.584162 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.584635 (+ 473us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.585486 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.585682 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.588123 (+ 2441us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:37.588152 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:37.588164 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:37.588195 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:37.590103 (+ 1908us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.590573 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.590579 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.590581 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.590640 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.590888 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.590891 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.590892 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.591156 (+ 264us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:37.591324 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.591511 (+ 187us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.591619 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 19:59:37.617205 28782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.609549 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36032 (local address 127.27.17.193:35195)
0429 19:59:37.609750 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.609755 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.609788 (+ 33us) server_negotiation.cc:408] Connection header received
0429 19:59:37.609968 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.609971 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.610034 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.610114 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:37.611055 (+ 941us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.611525 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.612249 (+ 724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.612394 (+ 145us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.614523 (+ 2129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:37.614546 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:37.614548 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:37.614574 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:37.615941 (+ 1367us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.616283 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.616286 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.616287 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.616324 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.616500 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.616503 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.616504 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.616619 (+ 115us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:37.616723 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.616917 (+ 194us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.617006 (+ 89us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 19:59:37.641993 28782 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:37.633766 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.27.17.193:35195)
0429 19:59:37.633970 (+ 204us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:37.633975 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:37.634080 (+ 105us) server_negotiation.cc:408] Connection header received
0429 19:59:37.634253 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:37.634256 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:37.634309 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:37.634397 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:37.635237 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.635706 (+ 469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:37.636353 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:37.636565 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:37.638891 (+ 2326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:37.638973 (+ 82us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:37.638976 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:37.639009 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:37.640414 (+ 1405us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.640876 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.640880 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.640882 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.640936 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:37.641192 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:37.641195 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:37.641198 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:37.641339 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:37.641472 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:37.641697 (+ 225us) server_negotiation.cc:300] Negotiation successful
0429 19:59:37.641824 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260429 19:59:37.643501 28408 tablet_service.cc:3018] Rejecting scan request for tablet 401c688fab2647abbff21ca62886fc1f: Uninitialized: clean time has not yet been initialized
I20260429 19:59:37.645358 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28365
I20260429 19:59:37.651782 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28501
I20260429 19:59:37.658030 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28637
I20260429 19:59:37.664163 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28269
2026-04-29T19:59:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3494 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28807](info): set up 2 sockets
Apr 29 19:59:37 dist-test-slave-01nr krb5kdc[28807](info): commencing operation
krb5kdc: starting...
W20260429 19:59:39.672693 27719 mini_kdc.cc:121] Time spent starting KDC: real 1.985s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 19:59:39 dist-test-slave-01nr krb5kdc[28807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T19:59:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T19:59:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:39.850965 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:36745
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:39.957820 28823 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:39.958127 28823 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:39.958236 28823 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:39.961767 28823 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 19:59:39.961875 28823 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 19:59:39.961913 28823 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:39.961987 28823 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:39.962047 28823 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 19:59:39.962092 28823 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 19:59:39.966638 28823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:36745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:39.967823 28823 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:39.968704 28823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:39.974280 28831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:39.974267 28828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:39.974269 28829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:39.974637 28823 server_base.cc:1061] running on GCE node
I20260429 19:59:39.975246 28823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:39.976254 28823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:39.977448 28823 hybrid_clock.cc:648] HybridClock initialized: now 1777492779977425 us; error 49 us; skew 500 ppm
Apr 29 19:59:39 dist-test-slave-01nr krb5kdc[28807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492779, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:39.980185 28823 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 19:59:39.981302 28823 webserver.cc:492] Webserver started at http://127.27.17.254:32823/ using document root <none> and password file <none>
I20260429 19:59:39.981869 28823 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:39.981949 28823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:39.982175 28823 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:39.983980 28823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "78f2faffae214ac681a14e7d8353eea6"
format_stamp: "Formatted at 2026-04-29 19:59:39 on dist-test-slave-01nr"
server_key: "b3b7fad56c8ecdcdc6626af43546ea72"
server_key_iv: "b503ad39e43cc42c8947ec9f397e4251"
server_key_version: "encryptionkey@0"
I20260429 19:59:39.984542 28823 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "78f2faffae214ac681a14e7d8353eea6"
format_stamp: "Formatted at 2026-04-29 19:59:39 on dist-test-slave-01nr"
server_key: "b3b7fad56c8ecdcdc6626af43546ea72"
server_key_iv: "b503ad39e43cc42c8947ec9f397e4251"
server_key_version: "encryptionkey@0"
I20260429 19:59:39.987995 28823 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 19:59:39.990206 28838 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:39.991303 28823 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 19:59:39.991458 28823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "78f2faffae214ac681a14e7d8353eea6"
format_stamp: "Formatted at 2026-04-29 19:59:39 on dist-test-slave-01nr"
server_key: "b3b7fad56c8ecdcdc6626af43546ea72"
server_key_iv: "b503ad39e43cc42c8947ec9f397e4251"
server_key_version: "encryptionkey@0"
I20260429 19:59:39.991582 28823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:40.007335 28823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:40.010722 28823 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:40.011021 28823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:40.019897 28823 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:36745
I20260429 19:59:40.020216 28900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:36745 every 8 connection(s)
I20260429 19:59:40.021099 28823 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 19:59:40.024047 28901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:40.027139 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28823
I20260429 19:59:40.027251 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 19:59:40.027495 27719 external_mini_cluster.cc:1468] Setting key 999dd0ff46a4e7e7ec4840de1f6cc058
I20260429 19:59:40.029798 28901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: Bootstrap starting.
I20260429 19:59:40.032044 28901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:40.032752 28901 log.cc:826] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:40.034701 28901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: No bootstrap required, opened a new log
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:40.037618 28901 raft_consensus.cc:359] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f2faffae214ac681a14e7d8353eea6" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36745 } }
I20260429 19:59:40.037842 28901 raft_consensus.cc:385] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:40.037945 28901 raft_consensus.cc:740] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f2faffae214ac681a14e7d8353eea6, State: Initialized, Role: FOLLOWER
I20260429 19:59:40.038365 28901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f2faffae214ac681a14e7d8353eea6" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36745 } }
I20260429 19:59:40.038522 28901 raft_consensus.cc:399] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 19:59:40.038606 28901 raft_consensus.cc:493] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 19:59:40.038710 28901 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:40.039806 28901 raft_consensus.cc:515] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f2faffae214ac681a14e7d8353eea6" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36745 } }
I20260429 19:59:40.040196 28901 leader_election.cc:304] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78f2faffae214ac681a14e7d8353eea6; no voters:
I20260429 19:59:40.040597 28901 leader_election.cc:290] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 19:59:40.040741 28906 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:40.040992 28906 raft_consensus.cc:697] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [term 1 LEADER]: Becoming Leader. State: Replica: 78f2faffae214ac681a14e7d8353eea6, State: Running, Role: LEADER
I20260429 19:59:40.041396 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.028903 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.27.17.254:36745)
0429 19:59:40.029373 (+ 470us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.029384 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.029419 (+ 35us) server_negotiation.cc:408] Connection header received
0429 19:59:40.030178 (+ 759us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.030203 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.030503 (+ 300us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.030917 (+ 414us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.031820 (+ 903us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.032632 (+ 812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.033352 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.033668 (+ 316us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.035909 (+ 2241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.035929 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.035945 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.035974 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.038462 (+ 2488us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.039032 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.039037 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.039042 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.039113 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.039366 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.039368 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.039369 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.039667 (+ 298us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.039799 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.040221 (+ 422us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.040500 (+ 279us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260429 19:59:40.041347 28906 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f2faffae214ac681a14e7d8353eea6" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36745 } }
I20260429 19:59:40.041826 28901 sys_catalog.cc:565] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 19:59:40.042503 28906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78f2faffae214ac681a14e7d8353eea6. Latest consensus state: current_term: 1 leader_uuid: "78f2faffae214ac681a14e7d8353eea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f2faffae214ac681a14e7d8353eea6" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36745 } } }
I20260429 19:59:40.042621 28906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:40.043159 28907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78f2faffae214ac681a14e7d8353eea6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78f2faffae214ac681a14e7d8353eea6" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36745 } } }
I20260429 19:59:40.043356 28907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6 [sys.catalog]: This master's current role is: LEADER
W20260429 19:59:40.046483 28921 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 19:59:40.046556 28921 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 19:59:40.046636 28915 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 19:59:40.047431 28915 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 19:59:40.052372 28915 catalog_manager.cc:1357] Generated new cluster ID: d69499f3f740472ba3f8b6821bbede47
I20260429 19:59:40.052459 28915 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 19:59:40.072743 28915 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 19:59:40.074028 28915 catalog_manager.cc:1514] Loading token signing keys...
I20260429 19:59:40.081015 28915 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: Generated new TSK 0
I20260429 19:59:40.081663 28915 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:40.151304 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:36745
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:40.259147 28929 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:40.259374 28929 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:40.259474 28929 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:40.263089 28929 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:40.263159 28929 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:40.263240 28929 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 19:59:40.263340 28929 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 19:59:40.267824 28929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:40.268983 28929 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:40.269887 28929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:40.276712 28937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:40.276784 28929 server_base.cc:1061] running on GCE node
W20260429 19:59:40.276706 28935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:40.276719 28934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:40.277354 28929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:40.277927 28929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:40.279086 28929 hybrid_clock.cc:648] HybridClock initialized: now 1777492780279059 us; error 38 us; skew 500 ppm
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:40.282148 28929 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 19:59:40.283499 28929 webserver.cc:492] Webserver started at http://127.27.17.193:43981/ using document root <none> and password file <none>
I20260429 19:59:40.284087 28929 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:40.284162 28929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:40.284375 28929 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:40.286154 28929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "941b95e7c4fa43a693f3c4ced6bb015e"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "5f92208e1b355922909b47a48544c118"
server_key_iv: "ff7647db21c41cb25dddbe8d8337980d"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.286652 28929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "941b95e7c4fa43a693f3c4ced6bb015e"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "5f92208e1b355922909b47a48544c118"
server_key_iv: "ff7647db21c41cb25dddbe8d8337980d"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.290153 28929 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 19:59:40.292574 28944 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:40.293785 28929 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 19:59:40.293964 28929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "941b95e7c4fa43a693f3c4ced6bb015e"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "5f92208e1b355922909b47a48544c118"
server_key_iv: "ff7647db21c41cb25dddbe8d8337980d"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.294088 28929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:40.330482 28929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:40.333449 28929 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:40.333676 28929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:40.335327 28929 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:40.335405 28929 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:40.335494 28929 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:40.335536 28929 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:40.347332 28929 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:39845
I20260429 19:59:40.347371 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.336043 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:36953 (local address 127.27.17.254:36745)
0429 19:59:40.336177 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.336180 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.337057 (+ 877us) server_negotiation.cc:408] Connection header received
0429 19:59:40.337982 (+ 925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.337986 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.338060 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.338166 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.339638 (+ 1472us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.340176 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.340839 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.341028 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.343571 (+ 2543us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.343592 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.343594 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.343622 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.345415 (+ 1793us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.345919 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.345923 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.345926 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.345981 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.346271 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.346275 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.346278 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.346433 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.346539 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.347072 (+ 533us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.347204 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 19:59:40.347353 29059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:39845 every 8 connection(s)
I20260429 19:59:40.347980 28953 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.336406 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.27.17.193:36953)
0429 19:59:40.336892 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.336926 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.337763 (+ 837us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.338354 (+ 591us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.338367 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.338956 (+ 589us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:40.339445 (+ 489us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.339456 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.340301 (+ 845us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.340306 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.340707 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.340713 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.340939 (+ 226us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.341648 (+ 709us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:40.341675 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:40.343342 (+ 1667us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:40.345566 (+ 2224us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.345574 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.345588 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.345807 (+ 219us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.346090 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.346093 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.346095 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.346179 (+ 84us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.346551 (+ 372us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:40.346566 (+ 15us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:40.346775 (+ 209us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.347036 (+ 261us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.347257 (+ 221us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
I20260429 19:59:40.348810 28929 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 19:59:40.350401 29060 heartbeater.cc:344] Connected to a master server at 127.27.17.254:36745
I20260429 19:59:40.350559 29060 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:40.351128 29060 heartbeater.cc:507] Master 127.27.17.254:36745 requested a full tablet report, sending...
I20260429 19:59:40.352768 28855 ts_manager.cc:194] Registered new tserver with Master: 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193:39845)
I20260429 19:59:40.354262 28855 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:36953
I20260429 19:59:40.358125 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 28929
I20260429 19:59:40.358268 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 19:59:40.358359 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.351595 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:47277 (local address 127.27.17.254:36745)
0429 19:59:40.351714 (+ 119us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.351717 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.352013 (+ 296us) server_negotiation.cc:408] Connection header received
0429 19:59:40.352120 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.352122 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.352165 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.352232 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.352916 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.353435 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.354266 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.354468 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.355354 (+ 886us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.355374 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.355381 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.355402 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.356859 (+ 1457us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.357289 (+ 430us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.357293 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.357295 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.357346 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.357627 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.357631 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.357633 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.357773 (+ 140us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.357885 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.358128 (+ 243us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.358229 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 19:59:40.358373 29066 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.351522 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.27.17.193:47277)
0429 19:59:40.351904 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.351938 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.352039 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.352271 (+ 232us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.352274 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.352477 (+ 203us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:40.352754 (+ 277us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.352763 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.353550 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.353554 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.354135 (+ 581us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.354143 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.354259 (+ 116us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.354808 (+ 549us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:40.354833 (+ 25us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:40.355229 (+ 396us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:40.357001 (+ 1772us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.357005 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.357007 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.357187 (+ 180us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.357440 (+ 253us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.357443 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.357445 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.357498 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.357875 (+ 377us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:40.357878 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:40.357961 (+ 83us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.358064 (+ 103us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.358191 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":226,"threads_started":1}
I20260429 19:59:40.358568 27719 external_mini_cluster.cc:1468] Setting key 75b80aa4311f7308bab16d8eaf6eeb32
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:40.412818 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:36745
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:40.519783 29071 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:40.520018 29071 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:40.520081 29071 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:40.523506 29071 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:40.523578 29071 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:40.523638 29071 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 19:59:40.523682 29071 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 19:59:40.528072 29071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:40.529165 29071 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:40.529995 29071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:40.536747 29079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:40.536773 29076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:40.536943 29071 server_base.cc:1061] running on GCE node
W20260429 19:59:40.536943 29077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:40.537403 29071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:40.537979 29071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:40.539132 29071 hybrid_clock.cc:648] HybridClock initialized: now 1777492780539106 us; error 40 us; skew 500 ppm
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:40.542184 29071 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 19:59:40.543467 29071 webserver.cc:492] Webserver started at http://127.27.17.194:34527/ using document root <none> and password file <none>
I20260429 19:59:40.544054 29071 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:40.544106 29071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:40.544449 29071 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:40.546121 29071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "07ace05f608c1c32e3a30e630326c756"
server_key_iv: "43f803c59d4c8c2858cd59ec3e26d33e"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.546541 29071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "07ace05f608c1c32e3a30e630326c756"
server_key_iv: "43f803c59d4c8c2858cd59ec3e26d33e"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.549912 29071 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 19:59:40.552028 29086 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:40.552983 29071 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:40.553112 29071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "07ace05f608c1c32e3a30e630326c756"
server_key_iv: "43f803c59d4c8c2858cd59ec3e26d33e"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.553215 29071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:40.580029 29071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:40.582976 29071 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:40.583180 29071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:40.584647 29071 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:40.584723 29071 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:40.584802 29071 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:40.584870 29071 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:40.597424 29201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:36829 every 8 connection(s)
I20260429 19:59:40.597398 29071 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:36829
I20260429 19:59:40.597795 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.585355 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:40751 (local address 127.27.17.254:36745)
0429 19:59:40.585498 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.585502 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.586320 (+ 818us) server_negotiation.cc:408] Connection header received
0429 19:59:40.587276 (+ 956us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.587280 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.587344 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.587450 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.589129 (+ 1679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.590142 (+ 1013us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.590941 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.591122 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.593568 (+ 2446us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.593599 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.593602 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.593633 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.595473 (+ 1840us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.596116 (+ 643us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.596123 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.596127 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.596192 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.596598 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.596602 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.596604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.596771 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.596887 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.597499 (+ 612us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.597630 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 19:59:40.598485 29095 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.585723 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.27.17.194:40751)
0429 19:59:40.586173 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.586207 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.587073 (+ 866us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.587655 (+ 582us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.587666 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.588237 (+ 571us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:40.588975 (+ 738us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.588988 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.590288 (+ 1300us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.590299 (+ 11us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.590793 (+ 494us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.590801 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.591052 (+ 251us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.591710 (+ 658us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:40.591730 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:40.593399 (+ 1669us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:40.595617 (+ 2218us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.595624 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.595640 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.595986 (+ 346us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.596336 (+ 350us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.596340 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.596343 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.596494 (+ 151us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.596891 (+ 397us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:40.596899 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:40.597193 (+ 294us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.597447 (+ 254us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.597703 (+ 256us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
I20260429 19:59:40.598593 29071 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 19:59:40.599287 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 29071
I20260429 19:59:40.599396 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 19:59:40.599664 27719 external_mini_cluster.cc:1468] Setting key 2d86ca754aa63618c9892449290ced7c
I20260429 19:59:40.600641 29202 heartbeater.cc:344] Connected to a master server at 127.27.17.254:36745
I20260429 19:59:40.600890 29202 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:40.601450 29202 heartbeater.cc:507] Master 127.27.17.254:36745 requested a full tablet report, sending...
I20260429 19:59:40.602593 28854 ts_manager.cc:194] Registered new tserver with Master: ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829)
I20260429 19:59:40.603404 28854 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:40751
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 19:59:40.610083 29209 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.603167 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.27.17.194:59441)
0429 19:59:40.603622 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.603638 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.603788 (+ 150us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.604080 (+ 292us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.604084 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.604276 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:40.604481 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.604487 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.605268 (+ 781us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.605273 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.605787 (+ 514us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.605797 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.605925 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.606464 (+ 539us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:40.606486 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:40.606980 (+ 494us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:40.608679 (+ 1699us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.608686 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.608690 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.608927 (+ 237us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.609196 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.609200 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.609202 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.609270 (+ 68us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.609667 (+ 397us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:40.609669 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:40.609745 (+ 76us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.609834 (+ 89us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.609941 (+ 107us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":88,"threads_started":1}
I20260429 19:59:40.610219 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.603162 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:59441 (local address 127.27.17.254:36745)
0429 19:59:40.603310 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.603315 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.603715 (+ 400us) server_negotiation.cc:408] Connection header received
0429 19:59:40.603904 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.603908 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.603960 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.604065 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.604600 (+ 535us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.605158 (+ 558us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.605925 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.606069 (+ 144us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.607106 (+ 1037us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.607120 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.607122 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.607145 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.608549 (+ 1404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.609046 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.609050 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.609051 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.609095 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.609377 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.609381 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.609383 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.609554 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.609656 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.609938 (+ 282us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.610047 (+ 109us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:40.659005 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:36745
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:40.762830 29213 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:40.763147 29213 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:40.763266 29213 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:40.766666 29213 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:40.766783 29213 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:40.766870 29213 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 19:59:40.766985 29213 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 19:59:40.771397 29213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:40.772521 29213 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:40.773416 29213 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:40.780237 29218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:40.780259 29221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:40.780237 29219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:40.780578 29213 server_base.cc:1061] running on GCE node
I20260429 19:59:40.780954 29213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:40.781553 29213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:40.782737 29213 hybrid_clock.cc:648] HybridClock initialized: now 1777492780782706 us; error 44 us; skew 500 ppm
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:40.785964 29213 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 19:59:40.787042 29213 webserver.cc:492] Webserver started at http://127.27.17.195:34647/ using document root <none> and password file <none>
I20260429 19:59:40.787582 29213 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:40.787629 29213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:40.787791 29213 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:40.789417 29213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "9c0316a582d84fdd9bae427a7828f611"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "0fe126214056f65d6a882952d32362f3"
server_key_iv: "78bb89065a0034fa7b97e5c50c183add"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.789873 29213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "9c0316a582d84fdd9bae427a7828f611"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "0fe126214056f65d6a882952d32362f3"
server_key_iv: "78bb89065a0034fa7b97e5c50c183add"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.793220 29213 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260429 19:59:40.795569 29228 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:40.796767 29213 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:40.796919 29213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "9c0316a582d84fdd9bae427a7828f611"
format_stamp: "Formatted at 2026-04-29 19:59:40 on dist-test-slave-01nr"
server_key: "0fe126214056f65d6a882952d32362f3"
server_key_iv: "78bb89065a0034fa7b97e5c50c183add"
server_key_version: "encryptionkey@0"
I20260429 19:59:40.797032 29213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:40.827731 29213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:40.830721 29213 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:40.830963 29213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:40.832770 29213 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:40.832828 29213 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:40.832875 29213 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:40.832952 29213 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:40.844497 29213 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:37625
I20260429 19:59:40.844594 29343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:37625 every 8 connection(s)
I20260429 19:59:40.845793 29213 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 19:59:40.847559 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.833723 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:33617 (local address 127.27.17.254:36745)
0429 19:59:40.833866 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.833870 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.834731 (+ 861us) server_negotiation.cc:408] Connection header received
0429 19:59:40.836118 (+ 1387us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.836123 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.836196 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.836263 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.838321 (+ 2058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.839217 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.839901 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.840082 (+ 181us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.843618 (+ 3536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.843647 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.843650 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.843684 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.845538 (+ 1854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.846099 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.846107 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.846111 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.846176 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.846545 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.846548 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.846550 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.846766 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.846877 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.847334 (+ 457us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.847430 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 19:59:40.848323 29237 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.834069 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.27.17.195:33617)
0429 19:59:40.834579 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.834613 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.835853 (+ 1240us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.836766 (+ 913us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.836780 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.837387 (+ 607us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:40.838160 (+ 773us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.838180 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.839356 (+ 1176us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.839360 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.839771 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.839777 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.840009 (+ 232us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.840696 (+ 687us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:40.840729 (+ 33us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:40.843429 (+ 2700us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:40.845673 (+ 2244us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.845678 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.845687 (+ 9us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.845994 (+ 307us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.846303 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.846307 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.846309 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.846429 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.846873 (+ 444us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:40.846877 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:40.847114 (+ 237us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.847280 (+ 166us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.847462 (+ 182us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":159,"threads_started":1}
I20260429 19:59:40.849676 29344 heartbeater.cc:344] Connected to a master server at 127.27.17.254:36745
I20260429 19:59:40.849866 29344 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:40.850399 29344 heartbeater.cc:507] Master 127.27.17.254:36745 requested a full tablet report, sending...
I20260429 19:59:40.851565 28855 ts_manager.cc:194] Registered new tserver with Master: 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625)
I20260429 19:59:40.852304 28855 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:33617
I20260429 19:59:40.855598 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 29213
I20260429 19:59:40.855713 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 19:59:40.855988 27719 external_mini_cluster.cc:1468] Setting key 25cb0c0b6a7cdc7740a20378f90948d9
I20260429 19:59:40.858290 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 19:59:40.859377 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.852173 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:33903 (local address 127.27.17.254:36745)
0429 19:59:40.852314 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.852321 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.852584 (+ 263us) server_negotiation.cc:408] Connection header received
0429 19:59:40.852707 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.852710 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.852757 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.852856 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.853579 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.854065 (+ 486us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.854725 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.854885 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.855831 (+ 946us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.855853 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.855858 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.855889 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.857637 (+ 1748us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.858249 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.858252 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.858253 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.858295 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.858592 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.858600 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.858601 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.858736 (+ 135us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.858865 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.859136 (+ 271us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.859231 (+ 95us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 19:59:40.859396 29350 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.852119 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.27.17.195:33903)
0429 19:59:40.852458 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.852474 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.852598 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.852902 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.852905 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.853108 (+ 203us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:40.853397 (+ 289us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.853407 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.854186 (+ 779us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.854190 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.854597 (+ 407us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.854604 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.854699 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.855280 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:40.855293 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:40.855678 (+ 385us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:40.857765 (+ 2087us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.857769 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.857770 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.857976 (+ 206us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.858436 (+ 460us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:40.858439 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:40.858440 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:40.858486 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:40.858855 (+ 369us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:40.858858 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:40.858973 (+ 115us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.859085 (+ 112us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.859221 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 19:59:40 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:40.883674 28904 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.875570 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.27.17.254:36745)
0429 19:59:40.875728 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.875732 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.875744 (+ 12us) server_negotiation.cc:408] Connection header received
0429 19:59:40.875782 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.875785 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.875847 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.875943 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:40.876787 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.877285 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.877945 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.878162 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.880474 (+ 2312us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:40.880495 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:40.880498 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:40.880577 (+ 79us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:40.881983 (+ 1406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.882396 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.882400 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.882401 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.882462 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:40.882710 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:40.882713 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:40.882715 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:40.882879 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:40.883081 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.883311 (+ 230us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.883476 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260429 19:59:40.886680 28855 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 19:59:40.889223 28855 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:40.902679 29361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.897958 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:39845 (local address 127.0.0.1:41864)
0429 19:59:40.898459 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.898498 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.898642 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.899426 (+ 784us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.899431 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.899454 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:40.899763 (+ 309us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.899770 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.901113 (+ 1343us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.901117 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.902184 (+ 1067us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.902193 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.902297 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.902352 (+ 55us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.902440 (+ 88us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.902520 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260429 19:59:40.903568 29366 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.899079 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60044 (local address 127.27.17.195:37625)
0429 19:59:40.899491 (+ 412us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.899500 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.899521 (+ 21us) server_negotiation.cc:408] Connection header received
0429 19:59:40.899603 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.899620 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.899772 (+ 152us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.899973 (+ 201us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:40.900384 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.901438 (+ 1054us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.902630 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.903184 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.903302 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.903366 (+ 64us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.903445 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":124,"threads_started":1}
I20260429 19:59:40.903899 29362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.898099 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41864 (local address 127.27.17.193:39845)
0429 19:59:40.898555 (+ 456us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.898562 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.898587 (+ 25us) server_negotiation.cc:408] Connection header received
0429 19:59:40.898884 (+ 297us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.898912 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.899113 (+ 201us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.899254 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:40.899912 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.900988 (+ 1076us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.903085 (+ 2097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.903491 (+ 406us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.903579 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.903675 (+ 96us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.903773 (+ 98us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":94,"threads_started":1}
I20260429 19:59:40.903949 29365 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.898989 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.27.17.194:36829)
0429 19:59:40.899351 (+ 362us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:40.899359 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:40.899707 (+ 348us) server_negotiation.cc:408] Connection header received
0429 19:59:40.899792 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:40.899798 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:40.899980 (+ 182us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:40.900118 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:40.900602 (+ 484us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.901684 (+ 1082us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.903035 (+ 1351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.903523 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.903625 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:40.903766 (+ 141us) server_negotiation.cc:300] Negotiation successful
0429 19:59:40.903841 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":110,"threads_started":1}
I20260429 19:59:40.902885 29363 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.898378 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:37625 (local address 127.0.0.1:60044)
0429 19:59:40.898760 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.898773 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.898864 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.899974 (+ 1110us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.899977 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.899997 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:40.900261 (+ 264us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.900267 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.901596 (+ 1329us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.901600 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.902469 (+ 869us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.902479 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.902623 (+ 144us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.902639 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.902690 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.902743 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":108,"threads_started":1}
I20260429 19:59:40.903085 29364 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:40.898664 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:36829 (local address 127.0.0.1:42564)
0429 19:59:40.899483 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:40.899497 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:40.899620 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:40.900261 (+ 641us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:40.900264 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:40.900281 (+ 17us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:40.900484 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.900489 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.901860 (+ 1371us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:40.901864 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:40.902703 (+ 839us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:40.902716 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:40.902837 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:40.902850 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 19:59:40.902917 (+ 67us) client_negotiation.cc:241] Negotiation successful
0429 19:59:40.902968 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":95,"threads_started":1}
I20260429 19:59:40.906996 29278 tablet_service.cc:1511] Processing CreateTablet for tablet d0cb3ae94c9248dc838c0b447976c32e (DEFAULT_TABLE table=test-table [id=0703b42ac6d14598ba1799c667c1a571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:40.907004 29136 tablet_service.cc:1511] Processing CreateTablet for tablet d0cb3ae94c9248dc838c0b447976c32e (DEFAULT_TABLE table=test-table [id=0703b42ac6d14598ba1799c667c1a571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:40.907003 28994 tablet_service.cc:1511] Processing CreateTablet for tablet d0cb3ae94c9248dc838c0b447976c32e (DEFAULT_TABLE table=test-table [id=0703b42ac6d14598ba1799c667c1a571]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:40.907922 29278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cb3ae94c9248dc838c0b447976c32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:40.907931 28994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cb3ae94c9248dc838c0b447976c32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:40.907931 29136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0cb3ae94c9248dc838c0b447976c32e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:40.913389 29367 tablet_bootstrap.cc:492] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Bootstrap starting.
I20260429 19:59:40.914650 29368 tablet_bootstrap.cc:492] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Bootstrap starting.
I20260429 19:59:40.915290 29369 tablet_bootstrap.cc:492] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Bootstrap starting.
I20260429 19:59:40.916332 29367 tablet_bootstrap.cc:654] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:40.916602 29368 tablet_bootstrap.cc:654] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:40.917301 29368 log.cc:826] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:40.917590 29367 log.cc:826] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:40.917629 29369 tablet_bootstrap.cc:654] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:40.918828 29369 log.cc:826] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:40.919566 29367 tablet_bootstrap.cc:492] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: No bootstrap required, opened a new log
I20260429 19:59:40.919798 29367 ts_tablet_manager.cc:1403] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260429 19:59:40.919812 29368 tablet_bootstrap.cc:492] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: No bootstrap required, opened a new log
I20260429 19:59:40.920063 29368 ts_tablet_manager.cc:1403] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260429 19:59:40.920677 29369 tablet_bootstrap.cc:492] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: No bootstrap required, opened a new log
I20260429 19:59:40.920871 29369 ts_tablet_manager.cc:1403] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260429 19:59:40.923110 29367 raft_consensus.cc:359] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:40.923368 29367 raft_consensus.cc:385] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:40.923456 29367 raft_consensus.cc:740] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0316a582d84fdd9bae427a7828f611, State: Initialized, Role: FOLLOWER
I20260429 19:59:40.923926 29367 consensus_queue.cc:260] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:40.923856 29369 raft_consensus.cc:359] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:40.923858 29368 raft_consensus.cc:359] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:40.924077 29369 raft_consensus.cc:385] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:40.924077 29368 raft_consensus.cc:385] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:40.924135 29368 raft_consensus.cc:740] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941b95e7c4fa43a693f3c4ced6bb015e, State: Initialized, Role: FOLLOWER
I20260429 19:59:40.924149 29369 raft_consensus.cc:740] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae046f7b64ec4bf9b74a8c3740c1ca85, State: Initialized, Role: FOLLOWER
I20260429 19:59:40.924546 29369 consensus_queue.cc:260] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:40.924546 29368 consensus_queue.cc:260] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:40.924940 29367 ts_tablet_manager.cc:1434] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260429 19:59:40.925217 29202 heartbeater.cc:499] Master 127.27.17.254:36745 was elected leader, sending a full tablet report...
I20260429 19:59:40.925352 29369 ts_tablet_manager.cc:1434] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20260429 19:59:40.925352 29368 ts_tablet_manager.cc:1434] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260429 19:59:40.925496 29344 heartbeater.cc:499] Master 127.27.17.254:36745 was elected leader, sending a full tablet report...
I20260429 19:59:40.925657 29060 heartbeater.cc:499] Master 127.27.17.254:36745 was elected leader, sending a full tablet report...
W20260429 19:59:41.096797 29345 tablet.cc:2404] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 19:59:41.100364 29061 tablet.cc:2404] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 19:59:41.100884 29203 tablet.cc:2404] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 19:59:41.238282 29374 raft_consensus.cc:493] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:41.238529 29374 raft_consensus.cc:515] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.239789 29374 leader_election.cc:290] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829), 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625)
I20260429 19:59:41.243155 29377 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.240085 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:36829 (local address 127.27.17.193:53535)
0429 19:59:41.240480 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.240492 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.240605 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.240888 (+ 283us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.240892 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.240911 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:41.241208 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.241216 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.242016 (+ 800us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.242020 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.242677 (+ 657us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.242685 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.242799 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.242815 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.242879 (+ 64us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.242994 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":125,"threads_started":1}
I20260429 19:59:41.243336 29376 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.240084 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:37625 (local address 127.27.17.193:46365)
0429 19:59:41.240671 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.240684 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.240784 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.241049 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.241051 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.241066 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:41.241320 (+ 254us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.241327 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.242130 (+ 803us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.242133 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.242785 (+ 652us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.242800 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.242978 (+ 178us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.242995 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.243053 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.243144 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":35,"thread_start_us":122,"threads_started":1}
I20260429 19:59:41.243655 29365 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.240175 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:53535 (local address 127.27.17.194:36829)
0429 19:59:41.240320 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.240324 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.240599 (+ 275us) server_negotiation.cc:408] Connection header received
0429 19:59:41.240699 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.240707 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.240778 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.240856 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:41.241329 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.241873 (+ 544us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.242825 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.243318 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.243360 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.243424 (+ 64us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.243530 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 19:59:41.243700 29366 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.240184 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:46365 (local address 127.27.17.195:37625)
0429 19:59:41.240331 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.240336 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.240783 (+ 447us) server_negotiation.cc:408] Connection header received
0429 19:59:41.240891 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.240894 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.240943 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.241029 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:41.241446 (+ 417us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.242012 (+ 566us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.242956 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.243423 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.243461 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.243515 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.243572 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 19:59:41.244320 29156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0cb3ae94c9248dc838c0b447976c32e" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" is_pre_election: true
I20260429 19:59:41.244331 29298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0cb3ae94c9248dc838c0b447976c32e" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0316a582d84fdd9bae427a7828f611" is_pre_election: true
I20260429 19:59:41.244589 29156 raft_consensus.cc:2468] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 0.
I20260429 19:59:41.244590 29298 raft_consensus.cc:2468] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 0.
I20260429 19:59:41.245083 28948 leader_election.cc:304] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941b95e7c4fa43a693f3c4ced6bb015e, ae046f7b64ec4bf9b74a8c3740c1ca85; no voters:
I20260429 19:59:41.245368 29374 raft_consensus.cc:2804] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:41.245442 29374 raft_consensus.cc:493] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:41.245476 29374 raft_consensus.cc:3060] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.246446 29374 raft_consensus.cc:515] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.246852 29374 leader_election.cc:290] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 election: Requested vote from peers ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829), 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625)
I20260429 19:59:41.247241 29298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0cb3ae94c9248dc838c0b447976c32e" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0316a582d84fdd9bae427a7828f611"
I20260429 19:59:41.247273 29156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0cb3ae94c9248dc838c0b447976c32e" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85"
I20260429 19:59:41.247347 29298 raft_consensus.cc:3060] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.247418 29156 raft_consensus.cc:3060] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.248395 29298 raft_consensus.cc:2468] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 1.
I20260429 19:59:41.248420 29156 raft_consensus.cc:2468] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 1.
I20260429 19:59:41.248749 28945 leader_election.cc:304] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941b95e7c4fa43a693f3c4ced6bb015e, 9c0316a582d84fdd9bae427a7828f611; no voters:
I20260429 19:59:41.249100 29374 raft_consensus.cc:2804] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:41.249353 29374 raft_consensus.cc:697] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 LEADER]: Becoming Leader. State: Replica: 941b95e7c4fa43a693f3c4ced6bb015e, State: Running, Role: LEADER
I20260429 19:59:41.249753 29374 consensus_queue.cc:237] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.253216 28855 catalog_manager.cc:5671] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e reported cstate change: term changed from 0 to 1, leader changed from <none> to 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193). New cstate: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } health_report { overall_health: UNKNOWN } } }
I20260429 19:59:41.275872 29362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.272152 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.27.17.193:39845)
0429 19:59:41.272294 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.272297 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.272315 (+ 18us) server_negotiation.cc:408] Connection header received
0429 19:59:41.272803 (+ 488us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.272807 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.272872 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.272969 (+ 97us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:41.273361 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.273960 (+ 599us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.274771 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.274971 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.275027 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.275418 (+ 391us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.275514 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.275694 (+ 180us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.275748 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 19:59:41.281572 29156 raft_consensus.cc:1275] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Refusing update from remote peer 941b95e7c4fa43a693f3c4ced6bb015e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:41.281594 29298 raft_consensus.cc:1275] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Refusing update from remote peer 941b95e7c4fa43a693f3c4ced6bb015e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:41.282369 29378 consensus_queue.cc:1048] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:41.282554 29374 consensus_queue.cc:1048] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:41.300810 28853 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:41.300998 28853 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:41.303490 28853 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: Sending DeleteTablet for 3 replicas of tablet d0cb3ae94c9248dc838c0b447976c32e
I20260429 19:59:41.304344 29278 tablet_service.cc:1558] Processing DeleteTablet for tablet d0cb3ae94c9248dc838c0b447976c32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:41 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:60044
I20260429 19:59:41.304349 29136 tablet_service.cc:1558] Processing DeleteTablet for tablet d0cb3ae94c9248dc838c0b447976c32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:41 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:42564
I20260429 19:59:41.304945 28994 tablet_service.cc:1558] Processing DeleteTablet for tablet d0cb3ae94c9248dc838c0b447976c32e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:41 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:41864
I20260429 19:59:41.305003 29392 tablet_replica.cc:333] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: stopping tablet replica
I20260429 19:59:41.305017 28853 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 19:59:41.305274 29392 raft_consensus.cc:2243] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260429 19:59:41.305439 28853 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:41.305608 29392 raft_consensus.cc:2272] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:41.305629 29393 tablet_replica.cc:333] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: stopping tablet replica
I20260429 19:59:41.305989 29391 tablet_replica.cc:333] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: stopping tablet replica
I20260429 19:59:41.305976 29393 raft_consensus.cc:2243] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 LEADER]: Raft consensus shutting down.
I20260429 19:59:41.306298 29391 raft_consensus.cc:2243] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:41.306394 29393 raft_consensus.cc:2272] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:41.307783 29392 ts_tablet_manager.cc:1916] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:41.307783 29393 ts_tablet_manager.cc:1916] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:41.310380 29393 ts_tablet_manager.cc:1929] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 19:59:41.310380 29392 ts_tablet_manager.cc:1929] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 19:59:41.310479 29392 log.cc:1199] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/d0cb3ae94c9248dc838c0b447976c32e
I20260429 19:59:41.310812 29392 ts_tablet_manager.cc:1950] T d0cb3ae94c9248dc838c0b447976c32e P 9c0316a582d84fdd9bae427a7828f611: Deleting consensus metadata
I20260429 19:59:41.310812 29391 raft_consensus.cc:2272] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:41.311388 28994 tablet_service.cc:1511] Processing CreateTablet for tablet e3cf2b0a77984e9487208888d49bc2ad (DEFAULT_TABLE table=test-table [id=6d85c3a93f1c43a5b280fc777e07b59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:41.311401 29136 tablet_service.cc:1511] Processing CreateTablet for tablet e3cf2b0a77984e9487208888d49bc2ad (DEFAULT_TABLE table=test-table [id=6d85c3a93f1c43a5b280fc777e07b59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:41.311659 29136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cf2b0a77984e9487208888d49bc2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:41.311677 28994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cf2b0a77984e9487208888d49bc2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:41.313560 29369 tablet_bootstrap.cc:492] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Bootstrap starting.
I20260429 19:59:41.313824 29393 log.cc:1199] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/d0cb3ae94c9248dc838c0b447976c32e
I20260429 19:59:41.314069 29368 tablet_bootstrap.cc:492] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Bootstrap starting.
I20260429 19:59:41.314167 29393 ts_tablet_manager.cc:1950] T d0cb3ae94c9248dc838c0b447976c32e P 941b95e7c4fa43a693f3c4ced6bb015e: Deleting consensus metadata
I20260429 19:59:41.314695 29369 tablet_bootstrap.cc:654] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:41.315094 29368 tablet_bootstrap.cc:654] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:41.315908 29369 tablet_bootstrap.cc:492] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: No bootstrap required, opened a new log
I20260429 19:59:41.315994 29369 ts_tablet_manager.cc:1403] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:41.316164 28842 catalog_manager.cc:5002] TS 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193:39845): tablet d0cb3ae94c9248dc838c0b447976c32e (table test-table [id=0703b42ac6d14598ba1799c667c1a571]) successfully deleted
I20260429 19:59:41.316459 29391 ts_tablet_manager.cc:1916] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:41.316464 28839 catalog_manager.cc:5002] TS 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625): tablet d0cb3ae94c9248dc838c0b447976c32e (table test-table [id=0703b42ac6d14598ba1799c667c1a571]) successfully deleted
I20260429 19:59:41.316438 29369 raft_consensus.cc:359] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.316563 29369 raft_consensus.cc:385] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:41.316596 29369 raft_consensus.cc:740] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae046f7b64ec4bf9b74a8c3740c1ca85, State: Initialized, Role: FOLLOWER
I20260429 19:59:41.316718 29369 consensus_queue.cc:260] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.316973 29369 ts_tablet_manager.cc:1434] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:41.316975 29278 tablet_service.cc:1511] Processing CreateTablet for tablet e3cf2b0a77984e9487208888d49bc2ad (DEFAULT_TABLE table=test-table [id=6d85c3a93f1c43a5b280fc777e07b59b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:41.317225 29278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cf2b0a77984e9487208888d49bc2ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:41.318848 29391 ts_tablet_manager.cc:1929] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 19:59:41.318994 29391 log.cc:1199] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/d0cb3ae94c9248dc838c0b447976c32e
I20260429 19:59:41.319370 29367 tablet_bootstrap.cc:492] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Bootstrap starting.
I20260429 19:59:41.319412 29391 ts_tablet_manager.cc:1950] T d0cb3ae94c9248dc838c0b447976c32e P ae046f7b64ec4bf9b74a8c3740c1ca85: Deleting consensus metadata
I20260429 19:59:41.320327 29367 tablet_bootstrap.cc:654] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:41.320410 28842 catalog_manager.cc:5002] TS ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829): tablet d0cb3ae94c9248dc838c0b447976c32e (table test-table [id=0703b42ac6d14598ba1799c667c1a571]) successfully deleted
I20260429 19:59:41.321136 29368 tablet_bootstrap.cc:492] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: No bootstrap required, opened a new log
I20260429 19:59:41.321242 29368 ts_tablet_manager.cc:1403] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260429 19:59:41.321664 29367 tablet_bootstrap.cc:492] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: No bootstrap required, opened a new log
I20260429 19:59:41.321650 29368 raft_consensus.cc:359] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.321738 29368 raft_consensus.cc:385] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:41.321738 29367 ts_tablet_manager.cc:1403] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:41.321761 29368 raft_consensus.cc:740] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941b95e7c4fa43a693f3c4ced6bb015e, State: Initialized, Role: FOLLOWER
I20260429 19:59:41.321908 29368 consensus_queue.cc:260] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.322167 29368 ts_tablet_manager.cc:1434] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 19:59:41.322180 29367 raft_consensus.cc:359] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.322273 29367 raft_consensus.cc:385] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:41.322304 29367 raft_consensus.cc:740] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0316a582d84fdd9bae427a7828f611, State: Initialized, Role: FOLLOWER
I20260429 19:59:41.322422 29367 consensus_queue.cc:260] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.322666 29367 ts_tablet_manager.cc:1434] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:41.447412 29373 raft_consensus.cc:493] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:41.447610 29373 raft_consensus.cc:515] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.448962 29373 leader_election.cc:290] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193:39845), ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829)
I20260429 19:59:41.452772 29397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.449193 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:39845 (local address 127.27.17.195:60747)
0429 19:59:41.449578 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.449595 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.449701 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.450163 (+ 462us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.450167 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.450193 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:41.450455 (+ 262us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.450465 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.451446 (+ 981us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.451451 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.452332 (+ 881us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.452341 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.452468 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.452485 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.452534 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.452591 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":92,"threads_started":1}
I20260429 19:59:41.453281 29362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.449360 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:60747 (local address 127.27.17.193:39845)
0429 19:59:41.449840 (+ 480us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.449845 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.449860 (+ 15us) server_negotiation.cc:408] Connection header received
0429 19:59:41.449914 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.449917 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.449976 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.450051 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:41.450649 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.451331 (+ 682us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.452484 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.453000 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.453040 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.453097 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.453163 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372}
I20260429 19:59:41.453708 29365 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.449851 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:39477 (local address 127.27.17.194:36829)
0429 19:59:41.450016 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.450021 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.450037 (+ 16us) server_negotiation.cc:408] Connection header received
0429 19:59:41.450236 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.450239 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.450290 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.450384 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:41.450850 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.451640 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.452714 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.453184 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.453227 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.453494 (+ 267us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.453561 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260429 19:59:41.453967 29014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3cf2b0a77984e9487208888d49bc2ad" candidate_uuid: "9c0316a582d84fdd9bae427a7828f611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" is_pre_election: true
I20260429 19:59:41.454063 29398 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.449585 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:36829 (local address 127.27.17.195:39477)
0429 19:59:41.449924 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.449984 (+ 60us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.450060 (+ 76us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.450397 (+ 337us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.450400 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.450419 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:41.450697 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.450705 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.451770 (+ 1065us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.451774 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.452578 (+ 804us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.452585 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.453294 (+ 709us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.453315 (+ 21us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.453849 (+ 534us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.453924 (+ 75us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":54,"threads_started":1}
I20260429 19:59:41.454206 29014 raft_consensus.cc:2468] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c0316a582d84fdd9bae427a7828f611 in term 0.
I20260429 19:59:41.454573 29156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3cf2b0a77984e9487208888d49bc2ad" candidate_uuid: "9c0316a582d84fdd9bae427a7828f611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" is_pre_election: true
I20260429 19:59:41.454720 29156 raft_consensus.cc:2468] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c0316a582d84fdd9bae427a7828f611 in term 0.
I20260429 19:59:41.454677 29232 leader_election.cc:304] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941b95e7c4fa43a693f3c4ced6bb015e, 9c0316a582d84fdd9bae427a7828f611; no voters:
I20260429 19:59:41.454932 29373 raft_consensus.cc:2804] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:41.454991 29373 raft_consensus.cc:493] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:41.455032 29373 raft_consensus.cc:3060] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.455955 29373 raft_consensus.cc:515] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.456301 29373 leader_election.cc:290] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [CANDIDATE]: Term 1 election: Requested vote from peers 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193:39845), ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829)
I20260429 19:59:41.456645 29014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3cf2b0a77984e9487208888d49bc2ad" candidate_uuid: "9c0316a582d84fdd9bae427a7828f611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "941b95e7c4fa43a693f3c4ced6bb015e"
I20260429 19:59:41.456772 29014 raft_consensus.cc:3060] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.456858 29156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e3cf2b0a77984e9487208888d49bc2ad" candidate_uuid: "9c0316a582d84fdd9bae427a7828f611" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85"
I20260429 19:59:41.457001 29156 raft_consensus.cc:3060] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.457957 29156 raft_consensus.cc:2468] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c0316a582d84fdd9bae427a7828f611 in term 1.
I20260429 19:59:41.457983 29014 raft_consensus.cc:2468] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c0316a582d84fdd9bae427a7828f611 in term 1.
I20260429 19:59:41.458271 29232 leader_election.cc:304] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941b95e7c4fa43a693f3c4ced6bb015e, 9c0316a582d84fdd9bae427a7828f611; no voters:
I20260429 19:59:41.458436 29373 raft_consensus.cc:2804] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:41.458680 29373 raft_consensus.cc:697] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 1 LEADER]: Becoming Leader. State: Replica: 9c0316a582d84fdd9bae427a7828f611, State: Running, Role: LEADER
I20260429 19:59:41.458968 29373 consensus_queue.cc:237] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.461431 28855 catalog_manager.cc:5671] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195). New cstate: current_term: 1 leader_uuid: "9c0316a582d84fdd9bae427a7828f611" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } health_report { overall_health: HEALTHY } } }
Apr 29 19:59:41 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492779, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:41.505950 29406 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.496995 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36745 (local address 127.0.0.1:38186)
0429 19:59:41.497330 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.497344 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.497449 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.497838 (+ 389us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.497842 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.498303 (+ 461us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:41.498518 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.498524 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.499354 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.499358 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.499766 (+ 408us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.499773 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.499881 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.500458 (+ 577us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:41.500471 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:41.502245 (+ 1774us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:41.504398 (+ 2153us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:41.504408 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:41.504414 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:41.504660 (+ 246us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:41.504915 (+ 255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:41.504920 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:41.504923 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:41.504972 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:41.505325 (+ 353us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:41.505333 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:41.505435 (+ 102us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.505567 (+ 132us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.505747 (+ 180us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
I20260429 19:59:41.505950 29407 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.497085 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.27.17.254:36745)
0429 19:59:41.497408 (+ 323us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.497411 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.497426 (+ 15us) server_negotiation.cc:408] Connection header received
0429 19:59:41.497597 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.497600 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.497647 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.497754 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:41.498675 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.499242 (+ 567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.499876 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.500011 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.502426 (+ 2415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:41.502447 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:41.502450 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:41.502474 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:41.504262 (+ 1788us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:41.504764 (+ 502us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:41.504769 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:41.504771 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:41.504828 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:41.505058 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:41.505062 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:41.505064 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:41.505217 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:41.505322 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.505616 (+ 294us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.505830 (+ 214us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":239,"thread_start_us":122,"threads_started":1}
I20260429 19:59:41.510252 28855 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:38186:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 19:59:41.511103 28855 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:41.516629 28994 tablet_service.cc:1511] Processing CreateTablet for tablet bd3b3131a5e242cb87688dd5f0139993 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b07f46d59ce440fb2d80a30eaf37c75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 19:59:41.516777 29136 tablet_service.cc:1511] Processing CreateTablet for tablet bd3b3131a5e242cb87688dd5f0139993 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b07f46d59ce440fb2d80a30eaf37c75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 19:59:41.516984 29278 tablet_service.cc:1511] Processing CreateTablet for tablet bd3b3131a5e242cb87688dd5f0139993 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2b07f46d59ce440fb2d80a30eaf37c75]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 19:59:41.517132 28994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3b3131a5e242cb87688dd5f0139993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:41.517169 29136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3b3131a5e242cb87688dd5f0139993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:41.517308 29278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3b3131a5e242cb87688dd5f0139993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:41.519533 29369 tablet_bootstrap.cc:492] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: Bootstrap starting.
I20260429 19:59:41.519973 29368 tablet_bootstrap.cc:492] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: Bootstrap starting.
I20260429 19:59:41.520368 29367 tablet_bootstrap.cc:492] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: Bootstrap starting.
I20260429 19:59:41.521387 29367 tablet_bootstrap.cc:654] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:41.521387 29369 tablet_bootstrap.cc:654] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:41.521518 29368 tablet_bootstrap.cc:654] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:41.522756 29369 tablet_bootstrap.cc:492] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: No bootstrap required, opened a new log
I20260429 19:59:41.522758 29367 tablet_bootstrap.cc:492] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: No bootstrap required, opened a new log
I20260429 19:59:41.522857 29369 ts_tablet_manager.cc:1403] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 19:59:41.522857 29367 ts_tablet_manager.cc:1403] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 19:59:41.523397 29368 tablet_bootstrap.cc:492] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: No bootstrap required, opened a new log
I20260429 19:59:41.523353 29367 raft_consensus.cc:359] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.523360 29369 raft_consensus.cc:359] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.523469 29369 raft_consensus.cc:385] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:41.523469 29367 raft_consensus.cc:385] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:41.523504 29367 raft_consensus.cc:740] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0316a582d84fdd9bae427a7828f611, State: Initialized, Role: FOLLOWER
I20260429 19:59:41.523504 29369 raft_consensus.cc:740] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae046f7b64ec4bf9b74a8c3740c1ca85, State: Initialized, Role: FOLLOWER
I20260429 19:59:41.523516 29368 ts_tablet_manager.cc:1403] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260429 19:59:41.523639 29369 consensus_queue.cc:260] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.523639 29367 consensus_queue.cc:260] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.523903 29369 ts_tablet_manager.cc:1434] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:41.523921 29367 ts_tablet_manager.cc:1434] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:41.524025 29368 raft_consensus.cc:359] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.524133 29368 raft_consensus.cc:385] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:41.524173 29368 raft_consensus.cc:740] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 941b95e7c4fa43a693f3c4ced6bb015e, State: Initialized, Role: FOLLOWER
I20260429 19:59:41.524128 29375 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.524260 29375 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.524302 29368 consensus_queue.cc:260] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.524565 29386 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.524703 29386 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.524574 29368 ts_tablet_manager.cc:1434] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:41.523910 29373 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.524979 29373 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.723380 29386 raft_consensus.cc:493] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:41.723549 29386 raft_consensus.cc:515] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.724082 29386 leader_election.cc:290] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829), 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625)
I20260429 19:59:41.724644 29156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3b3131a5e242cb87688dd5f0139993" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" is_pre_election: true
I20260429 19:59:41.724644 29298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3b3131a5e242cb87688dd5f0139993" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0316a582d84fdd9bae427a7828f611" is_pre_election: true
I20260429 19:59:41.724815 29156 raft_consensus.cc:2468] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 0.
I20260429 19:59:41.724840 29298 raft_consensus.cc:2468] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 0.
I20260429 19:59:41.725220 28948 leader_election.cc:304] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941b95e7c4fa43a693f3c4ced6bb015e, ae046f7b64ec4bf9b74a8c3740c1ca85; no voters:
I20260429 19:59:41.725450 29386 raft_consensus.cc:2804] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:41.725550 29386 raft_consensus.cc:493] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:41.725581 29386 raft_consensus.cc:3060] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.726720 29386 raft_consensus.cc:515] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.727161 29386 leader_election.cc:290] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 election: Requested vote from peers ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829), 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625)
I20260429 19:59:41.727624 29156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3b3131a5e242cb87688dd5f0139993" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85"
I20260429 19:59:41.727650 29298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd3b3131a5e242cb87688dd5f0139993" candidate_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0316a582d84fdd9bae427a7828f611"
I20260429 19:59:41.727770 29156 raft_consensus.cc:3060] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.727788 29298 raft_consensus.cc:3060] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:41.728652 29156 raft_consensus.cc:2468] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 1.
I20260429 19:59:41.728688 29298 raft_consensus.cc:2468] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 941b95e7c4fa43a693f3c4ced6bb015e in term 1.
I20260429 19:59:41.729020 28948 leader_election.cc:304] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 941b95e7c4fa43a693f3c4ced6bb015e, ae046f7b64ec4bf9b74a8c3740c1ca85; no voters:
I20260429 19:59:41.729214 29386 raft_consensus.cc:2804] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:41.729367 29386 raft_consensus.cc:697] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 LEADER]: Becoming Leader. State: Replica: 941b95e7c4fa43a693f3c4ced6bb015e, State: Running, Role: LEADER
I20260429 19:59:41.729533 29386 consensus_queue.cc:237] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } }
I20260429 19:59:41.730259 29378 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: TxnStatusTablet state changed. Reason: New leader 941b95e7c4fa43a693f3c4ced6bb015e. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.730373 29378 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: This TxnStatusTablet replica's current role is: LEADER
I20260429 19:59:41.730659 29412 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 19:59:41.730811 29412 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 19:59:41.731518 28855 catalog_manager.cc:5671] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e reported cstate change: term changed from 0 to 1, leader changed from <none> to 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193). New cstate: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } health_report { overall_health: UNKNOWN } } }
I20260429 19:59:41.766583 29406 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.763518 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:39845 (local address 127.0.0.1:41876)
0429 19:59:41.763682 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.763701 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.763819 (+ 118us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.764117 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.764119 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.764127 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:41.764307 (+ 180us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.764312 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.765350 (+ 1038us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.765354 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.765775 (+ 421us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.765781 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.765872 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.765952 (+ 80us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.766311 (+ 359us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.766321 (+ 10us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.766378 (+ 57us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.766438 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260429 19:59:41.766599 29362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.763596 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.27.17.193:39845)
0429 19:59:41.763746 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.763751 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.763768 (+ 17us) server_negotiation.cc:408] Connection header received
0429 19:59:41.763935 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.763939 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.763999 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.764087 (+ 88us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:41.764418 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.765213 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.765892 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.766088 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.766129 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.766211 (+ 82us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.766289 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.766420 (+ 131us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.766467 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 19:59:41.769001 29156 raft_consensus.cc:1275] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Refusing update from remote peer 941b95e7c4fa43a693f3c4ced6bb015e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:41.769019 29298 raft_consensus.cc:1275] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Refusing update from remote peer 941b95e7c4fa43a693f3c4ced6bb015e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:41.769508 29378 consensus_queue.cc:1048] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:41.769663 29386 consensus_queue.cc:1048] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:41.771152 29375 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: TxnStatusTablet state changed. Reason: New leader 941b95e7c4fa43a693f3c4ced6bb015e. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.771266 29375 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.771427 29373 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: TxnStatusTablet state changed. Reason: New leader 941b95e7c4fa43a693f3c4ced6bb015e. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.771528 29373 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.773355 29378 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.773463 29378 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: This TxnStatusTablet replica's current role is: LEADER
I20260429 19:59:41.773567 29381 mvcc.cc:204] Tried to move back new op lower bound from 7280610434121732096 to 7280610433965838336. Current Snapshot: MvccSnapshot[applied={T|T < 7280610434121732096}]
I20260429 19:59:41.775347 29382 mvcc.cc:204] Tried to move back new op lower bound from 7280610434121732096 to 7280610433965838336. Current Snapshot: MvccSnapshot[applied={T|T < 7280610434121732096}]
I20260429 19:59:41.775535 29375 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.775619 29375 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P ae046f7b64ec4bf9b74a8c3740c1ca85: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.777382 29374 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.777504 29374 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 941b95e7c4fa43a693f3c4ced6bb015e: This TxnStatusTablet replica's current role is: LEADER
I20260429 19:59:41.778337 29383 mvcc.cc:204] Tried to move back new op lower bound from 7280610434121732096 to 7280610433965838336. Current Snapshot: MvccSnapshot[applied={T|T < 7280610434121732096}]
I20260429 19:59:41.778544 29373 tablet_replica.cc:442] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 } } peers { permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 } } peers { permanent_uuid: "9c0316a582d84fdd9bae427a7828f611" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 37625 } } }
I20260429 19:59:41.778651 29373 tablet_replica.cc:445] T bd3b3131a5e242cb87688dd5f0139993 P 9c0316a582d84fdd9bae427a7828f611: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:41.783571 29419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.778977 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60052 (local address 127.27.17.195:37625)
0429 19:59:41.779522 (+ 545us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.779527 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.779540 (+ 13us) server_negotiation.cc:408] Connection header received
0429 19:59:41.779616 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.779620 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.779677 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.779794 (+ 117us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:41.780358 (+ 564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.781279 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.782240 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.782468 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.782542 (+ 74us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.783089 (+ 547us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.783197 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.783321 (+ 124us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.783376 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":100,"threads_started":1}
I20260429 19:59:41.794763 29362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.791441 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:36001 (local address 127.27.17.193:39845)
0429 19:59:41.791677 (+ 236us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.791681 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.791703 (+ 22us) server_negotiation.cc:408] Connection header received
0429 19:59:41.791882 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.791885 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.791946 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.792032 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:41.792384 (+ 352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.793132 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.793916 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.794123 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.794169 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.794264 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.794347 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.794563 (+ 216us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.794609 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260429 19:59:41.794785 29422 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.791324 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:39845 (local address 127.27.17.195:36001)
0429 19:59:41.791647 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.791661 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.791749 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.792047 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.792049 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.792055 (+ 6us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:41.792281 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.792286 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.793273 (+ 987us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.793277 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.793792 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.793801 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.793915 (+ 114us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.793956 (+ 41us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.794391 (+ 435us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.794398 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.794533 (+ 135us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.794607 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":118,"threads_started":1}
I20260429 19:59:41.802546 29156 raft_consensus.cc:1275] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Refusing update from remote peer 9c0316a582d84fdd9bae427a7828f611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:41.802700 29014 raft_consensus.cc:1275] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Refusing update from remote peer 9c0316a582d84fdd9bae427a7828f611: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:41.803113 29399 consensus_queue.cc:1048] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae046f7b64ec4bf9b74a8c3740c1ca85" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:41.803260 29373 consensus_queue.cc:1048] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [LEADER]: Connected to new peer: Peer: permanent_uuid: "941b95e7c4fa43a693f3c4ced6bb015e" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:41.827847 29426 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.823988 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:37625 (local address 127.27.17.193:33171)
0429 19:59:41.824320 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:41.824336 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:41.824484 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:41.824813 (+ 329us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:41.824816 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:41.824824 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:41.825037 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.825042 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.826097 (+ 1055us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.826102 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:41.826787 (+ 685us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:41.826796 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.826929 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.826978 (+ 49us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.827576 (+ 598us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.827583 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 19:59:41.827642 (+ 59us) client_negotiation.cc:241] Negotiation successful
0429 19:59:41.827703 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260429 19:59:41.827864 29419 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.824160 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:33171 (local address 127.27.17.195:37625)
0429 19:59:41.824294 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.824298 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.824413 (+ 115us) server_negotiation.cc:408] Connection header received
0429 19:59:41.824588 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.824592 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.824650 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.824736 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:41.825194 (+ 458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.825964 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.827101 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.827317 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.827360 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:41.827450 (+ 90us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:41.827531 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.827676 (+ 145us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.827725 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 19:59:41.852865 28855 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:41.853024 28855 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:41.855302 28855 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 78f2faffae214ac681a14e7d8353eea6: Sending DeleteTablet for 3 replicas of tablet e3cf2b0a77984e9487208888d49bc2ad
I20260429 19:59:41.856003 29136 tablet_service.cc:1558] Processing DeleteTablet for tablet e3cf2b0a77984e9487208888d49bc2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:41 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:42564
I20260429 19:59:41.856003 28994 tablet_service.cc:1558] Processing DeleteTablet for tablet e3cf2b0a77984e9487208888d49bc2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:41 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:41864
I20260429 19:59:41.856602 29430 tablet_replica.cc:333] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: stopping tablet replica
I20260429 19:59:41.856645 29278 tablet_service.cc:1558] Processing DeleteTablet for tablet e3cf2b0a77984e9487208888d49bc2ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:41 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:60044
I20260429 19:59:41.856879 29431 tablet_replica.cc:333] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: stopping tablet replica
I20260429 19:59:41.857028 29431 raft_consensus.cc:2243] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:41.857229 29430 raft_consensus.cc:2243] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:41.857399 29431 raft_consensus.cc:2272] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:41.857550 29430 raft_consensus.cc:2272] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:41.859066 29431 ts_tablet_manager.cc:1916] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:41.859617 29430 ts_tablet_manager.cc:1916] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:41.862362 29430 ts_tablet_manager.cc:1929] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:41.864955 29431 ts_tablet_manager.cc:1929] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:41.865777 29431 log.cc:1199] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/e3cf2b0a77984e9487208888d49bc2ad
I20260429 19:59:41.866140 29431 ts_tablet_manager.cc:1950] T e3cf2b0a77984e9487208888d49bc2ad P 941b95e7c4fa43a693f3c4ced6bb015e: Deleting consensus metadata
I20260429 19:59:41.866264 29430 log.cc:1199] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/e3cf2b0a77984e9487208888d49bc2ad
I20260429 19:59:41.866595 29430 ts_tablet_manager.cc:1950] T e3cf2b0a77984e9487208888d49bc2ad P ae046f7b64ec4bf9b74a8c3740c1ca85: Deleting consensus metadata
I20260429 19:59:41.867076 29432 tablet_replica.cc:333] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: stopping tablet replica
I20260429 19:59:41.867269 29432 raft_consensus.cc:2243] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 1 LEADER]: Raft consensus shutting down.
I20260429 19:59:41.867527 28842 catalog_manager.cc:5002] TS 941b95e7c4fa43a693f3c4ced6bb015e (127.27.17.193:39845): tablet e3cf2b0a77984e9487208888d49bc2ad (table test-table [id=6d85c3a93f1c43a5b280fc777e07b59b]) successfully deleted
I20260429 19:59:41.867681 29432 raft_consensus.cc:2272] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:41.868538 28842 catalog_manager.cc:5002] TS ae046f7b64ec4bf9b74a8c3740c1ca85 (127.27.17.194:36829): tablet e3cf2b0a77984e9487208888d49bc2ad (table test-table [id=6d85c3a93f1c43a5b280fc777e07b59b]) successfully deleted
I20260429 19:59:41.868952 29432 ts_tablet_manager.cc:1916] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:41.871919 29432 ts_tablet_manager.cc:1929] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:41.872062 29432 log.cc:1199] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/e3cf2b0a77984e9487208888d49bc2ad
I20260429 19:59:41.872473 29432 ts_tablet_manager.cc:1950] T e3cf2b0a77984e9487208888d49bc2ad P 9c0316a582d84fdd9bae427a7828f611: Deleting consensus metadata
I20260429 19:59:41.875727 28839 catalog_manager.cc:5002] TS 9c0316a582d84fdd9bae427a7828f611 (127.27.17.195:37625): tablet e3cf2b0a77984e9487208888d49bc2ad (table test-table [id=6d85c3a93f1c43a5b280fc777e07b59b]) successfully deleted
Apr 29 19:59:41 dist-test-slave-01nr krb5kdc[28807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492780, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 19:59:41.884711 29362 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.858394 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.27.17.193:39845)
0429 19:59:41.858535 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.858540 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.868759 (+ 10219us) server_negotiation.cc:408] Connection header received
0429 19:59:41.868989 (+ 230us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.868992 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.869057 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.869122 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:41.870390 (+ 1268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.871113 (+ 723us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.872072 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.872298 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.880377 (+ 8079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:41.880417 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:41.880434 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:41.880476 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:41.882725 (+ 2249us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:41.883386 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:41.883391 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:41.883392 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:41.883450 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:41.883753 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:41.883756 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:41.883760 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:41.883958 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:41.884161 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.884367 (+ 206us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.884502 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260429 19:59:41.885288 29054 server_base.cc:1143] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41890
I20260429 19:59:41.894613 29407 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:41.887009 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.27.17.254:36745)
0429 19:59:41.887151 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:41.887157 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:41.887742 (+ 585us) server_negotiation.cc:408] Connection header received
0429 19:59:41.887915 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:41.887919 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:41.887982 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:41.888073 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:41.888888 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.889421 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:41.890068 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:41.890358 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:41.891495 (+ 1137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:41.891529 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:41.891532 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:41.891561 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:41.893054 (+ 1493us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:41.893489 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:41.893492 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:41.893494 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:41.893544 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:41.893782 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:41.893786 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:41.893788 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:41.893949 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:41.894093 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:41.894306 (+ 213us) server_negotiation.cc:300] Negotiation successful
0429 19:59:41.894429 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260429 19:59:41.895260 28855 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38202
I20260429 19:59:41.896564 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28929
I20260429 19:59:41.905699 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 29071
I20260429 19:59:41.912700 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 29213
I20260429 19:59:41.920233 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 28823
2026-04-29T19:59:41Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4259 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:41 dist-test-slave-01nr krb5kdc[29439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:41 dist-test-slave-01nr krb5kdc[29439](info): set up 2 sockets
Apr 29 19:59:41 dist-test-slave-01nr krb5kdc[29439](info): commencing operation
krb5kdc: starting...
W20260429 19:59:43.948390 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T19:59:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T19:59:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:44.098876 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:35247
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:35247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:44.206020 29455 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:44.206359 29455 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:44.206497 29455 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:44.210050 29455 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 19:59:44.210175 29455 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 19:59:44.210211 29455 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:44.210258 29455 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:44.210318 29455 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 19:59:44.210368 29455 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 19:59:44.215145 29455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:35247
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:35247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:44.216385 29455 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:44.217310 29455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:44.223169 29463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:44.223161 29461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:44.223166 29460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:44.223508 29455 server_base.cc:1061] running on GCE node
I20260429 19:59:44.224135 29455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:44.225183 29455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:44.226416 29455 hybrid_clock.cc:648] HybridClock initialized: now 1777492784226391 us; error 46 us; skew 500 ppm
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:44.229655 29455 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 19:59:44.230967 29455 webserver.cc:492] Webserver started at http://127.27.17.254:39965/ using document root <none> and password file <none>
I20260429 19:59:44.231567 29455 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:44.231650 29455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:44.231874 29455 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:44.233667 29455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "4020d4127b8246a9bc6ff9352c5e30df"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "8d3186d28afd1bbf93e02ff34652e3e1"
server_key_iv: "ca77375e0ec5655a1cc9abe04efab6bf"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.234185 29455 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "4020d4127b8246a9bc6ff9352c5e30df"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "8d3186d28afd1bbf93e02ff34652e3e1"
server_key_iv: "ca77375e0ec5655a1cc9abe04efab6bf"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.237836 29455 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 19:59:44.240494 29470 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:44.241644 29455 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 19:59:44.241802 29455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "4020d4127b8246a9bc6ff9352c5e30df"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "8d3186d28afd1bbf93e02ff34652e3e1"
server_key_iv: "ca77375e0ec5655a1cc9abe04efab6bf"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.241925 29455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:44.271306 29455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:44.274457 29455 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:44.274709 29455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:44.284006 29455 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:35247
I20260429 19:59:44.284034 29532 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:35247 every 8 connection(s)
I20260429 19:59:44.285157 29455 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 19:59:44.285472 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 29455
I20260429 19:59:44.285599 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 19:59:44.285889 27719 external_mini_cluster.cc:1468] Setting key a71bacf8a0d73195b9ca05d96c78c9cb
I20260429 19:59:44.288509 29533 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:44.294111 29533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Bootstrap starting.
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:44.296854 29533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:44.297649 29533 log.cc:826] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:44.300016 29533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: No bootstrap required, opened a new log
I20260429 19:59:44.300619 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.287336 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.27.17.254:35247)
0429 19:59:44.288213 (+ 877us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:44.288224 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:44.288263 (+ 39us) server_negotiation.cc:408] Connection header received
0429 19:59:44.289075 (+ 812us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:44.289095 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:44.289417 (+ 322us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:44.289778 (+ 361us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:44.290836 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.291811 (+ 975us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.292577 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.292873 (+ 296us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.295445 (+ 2572us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:44.295492 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:44.295506 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:44.295543 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:44.297541 (+ 1998us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.298102 (+ 561us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.298107 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.298113 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.298198 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.298484 (+ 286us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.298488 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.298490 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.298854 (+ 364us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:44.299142 (+ 288us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:44.299445 (+ 303us) server_negotiation.cc:300] Negotiation successful
0429 19:59:44.299821 (+ 376us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":156,"threads_started":1}
I20260429 19:59:44.303503 29533 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4020d4127b8246a9bc6ff9352c5e30df" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35247 } }
I20260429 19:59:44.303785 29533 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:44.303864 29533 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4020d4127b8246a9bc6ff9352c5e30df, State: Initialized, Role: FOLLOWER
I20260429 19:59:44.304570 29533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4020d4127b8246a9bc6ff9352c5e30df" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35247 } }
I20260429 19:59:44.304792 29533 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 19:59:44.304877 29533 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 19:59:44.305006 29533 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:44.306440 29533 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4020d4127b8246a9bc6ff9352c5e30df" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35247 } }
I20260429 19:59:44.306954 29533 leader_election.cc:304] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4020d4127b8246a9bc6ff9352c5e30df; no voters:
I20260429 19:59:44.307425 29533 leader_election.cc:290] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 19:59:44.307610 29538 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:44.307861 29538 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [term 1 LEADER]: Becoming Leader. State: Replica: 4020d4127b8246a9bc6ff9352c5e30df, State: Running, Role: LEADER
I20260429 19:59:44.308259 29538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4020d4127b8246a9bc6ff9352c5e30df" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35247 } }
I20260429 19:59:44.308774 29533 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [sys.catalog]: configured and running, proceeding with master startup.
I20260429 19:59:44.310003 29540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4020d4127b8246a9bc6ff9352c5e30df. Latest consensus state: current_term: 1 leader_uuid: "4020d4127b8246a9bc6ff9352c5e30df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4020d4127b8246a9bc6ff9352c5e30df" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35247 } } }
I20260429 19:59:44.310138 29540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:44.310482 29539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4020d4127b8246a9bc6ff9352c5e30df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4020d4127b8246a9bc6ff9352c5e30df" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35247 } } }
I20260429 19:59:44.310671 29539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:44.310532 29547 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 19:59:44.314481 29547 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 19:59:44.320945 29547 catalog_manager.cc:1357] Generated new cluster ID: 5737219b61a64c85be9e0269e95f007e
I20260429 19:59:44.321069 29547 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 19:59:44.329902 29547 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 19:59:44.330988 29547 catalog_manager.cc:1514] Loading token signing keys...
I20260429 19:59:44.340919 29547 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Generated new TSK 0
I20260429 19:59:44.341691 29547 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:44.404373 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35247
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:44.527796 29561 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:44.528024 29561 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:44.528090 29561 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:44.531467 29561 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:44.531531 29561 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:44.531585 29561 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 19:59:44.531607 29561 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 19:59:44.531648 29561 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 19:59:44.536037 29561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:35247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:44.537098 29561 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:44.537923 29561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:44.544476 29569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:44.544550 29566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:44.544553 29567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:44.544862 29561 server_base.cc:1061] running on GCE node
I20260429 19:59:44.545212 29561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:44.545782 29561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:44.546975 29561 hybrid_clock.cc:648] HybridClock initialized: now 1777492784546956 us; error 30 us; skew 500 ppm
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:44.549721 29561 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 19:59:44.550822 29561 webserver.cc:492] Webserver started at http://127.27.17.193:34113/ using document root <none> and password file <none>
I20260429 19:59:44.551469 29561 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:44.551550 29561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:44.551776 29561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:44.553565 29561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "5fa60c1192024a6ca11d0c2561fc4619"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "312fe292aeaac67d24b1648599457642"
server_key_iv: "6f88678c6479460a29d45fb118893250"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.554078 29561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "5fa60c1192024a6ca11d0c2561fc4619"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "312fe292aeaac67d24b1648599457642"
server_key_iv: "6f88678c6479460a29d45fb118893250"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.557775 29561 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 19:59:44.560251 29576 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:44.561455 29561 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:44.561594 29561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "5fa60c1192024a6ca11d0c2561fc4619"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "312fe292aeaac67d24b1648599457642"
server_key_iv: "6f88678c6479460a29d45fb118893250"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.561708 29561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:44.583263 29561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:44.586715 29561 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:44.587000 29561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:44.588601 29561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:44.588681 29561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:44.588758 29561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:44.588800 29561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:44.600725 29561 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:44421
I20260429 19:59:44.600736 29691 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:44421 every 8 connection(s)
I20260429 19:59:44.601914 29561 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 19:59:44.602552 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.589625 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:49775 (local address 127.27.17.254:35247)
0429 19:59:44.589808 (+ 183us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:44.589814 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:44.590467 (+ 653us) server_negotiation.cc:408] Connection header received
0429 19:59:44.591495 (+ 1028us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:44.591500 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:44.591561 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:44.591644 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:44.593535 (+ 1891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.594354 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.595389 (+ 1035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.595635 (+ 246us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.598082 (+ 2447us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:44.598115 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:44.598119 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:44.598163 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:44.600164 (+ 2001us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.600886 (+ 722us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.600889 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.600891 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.600947 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.601268 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.601271 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.601273 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.601460 (+ 187us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:44.601589 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:44.602201 (+ 612us) server_negotiation.cc:300] Negotiation successful
0429 19:59:44.602368 (+ 167us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260429 19:59:44.603324 29585 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.589878 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.27.17.193:49775)
0429 19:59:44.590308 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:44.590348 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:44.591266 (+ 918us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:44.591926 (+ 660us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:44.591940 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:44.592582 (+ 642us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:44.593317 (+ 735us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.593339 (+ 22us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.594509 (+ 1170us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.594516 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:44.595235 (+ 719us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.595249 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.595515 (+ 266us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.596276 (+ 761us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:44.596305 (+ 29us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:44.597888 (+ 1583us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:44.600339 (+ 2451us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.600347 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.600362 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.600741 (+ 379us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.601053 (+ 312us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.601057 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.601059 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.601159 (+ 100us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.601612 (+ 453us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:44.601620 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:44.601894 (+ 274us) client_negotiation.cc:770] Sending connection context
0429 19:59:44.602123 (+ 229us) client_negotiation.cc:241] Negotiation successful
0429 19:59:44.602407 (+ 284us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":138,"threads_started":1}
I20260429 19:59:44.604637 29692 heartbeater.cc:344] Connected to a master server at 127.27.17.254:35247
I20260429 19:59:44.604825 29692 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:44.605376 29692 heartbeater.cc:507] Master 127.27.17.254:35247 requested a full tablet report, sending...
I20260429 19:59:44.607287 29486 ts_manager.cc:194] Registered new tserver with Master: 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421)
I20260429 19:59:44.608611 29486 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:49775
I20260429 19:59:44.611938 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 29561
I20260429 19:59:44.612082 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 19:59:44.612365 27719 external_mini_cluster.cc:1468] Setting key 1b05c8b88480ec570e9b4eafb36f5c68
I20260429 19:59:44.614121 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.606707 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:59975 (local address 127.27.17.254:35247)
0429 19:59:44.606860 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:44.606866 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:44.607097 (+ 231us) server_negotiation.cc:408] Connection header received
0429 19:59:44.607151 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:44.607153 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:44.607198 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:44.607281 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:44.607975 (+ 694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.608752 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.609520 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.609752 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.610795 (+ 1043us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:44.610825 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:44.610828 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:44.610852 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:44.612535 (+ 1683us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.612943 (+ 408us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.612947 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.612949 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.613005 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.613279 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.613282 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.613284 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.613461 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:44.613575 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:44.613844 (+ 269us) server_negotiation.cc:300] Negotiation successful
0429 19:59:44.613957 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 19:59:44.614226 29698 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.606589 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.27.17.193:59975)
0429 19:59:44.606961 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:44.606977 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:44.607080 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:44.607316 (+ 236us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:44.607322 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:44.607530 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:44.607835 (+ 305us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.607843 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.608888 (+ 1045us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.608894 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:44.609394 (+ 500us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.609403 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.609534 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.610228 (+ 694us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:44.610242 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:44.610660 (+ 418us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:44.612652 (+ 1992us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.612655 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.612657 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.612838 (+ 181us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.613098 (+ 260us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.613102 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.613104 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.613186 (+ 82us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.613596 (+ 410us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:44.613600 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:44.613678 (+ 78us) client_negotiation.cc:770] Sending connection context
0429 19:59:44.613767 (+ 89us) client_negotiation.cc:241] Negotiation successful
0429 19:59:44.614017 (+ 250us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:44.671672 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35247
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:44.786813 29703 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:44.787238 29703 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:44.787384 29703 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:44.791262 29703 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:44.791414 29703 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:44.791517 29703 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 19:59:44.791566 29703 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 19:59:44.791648 29703 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 19:59:44.796677 29703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:35247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:44.798029 29703 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:44.799067 29703 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:44.806674 29708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:44.806629 29711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:44.806629 29709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:44.807157 29703 server_base.cc:1061] running on GCE node
I20260429 19:59:44.807585 29703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:44.808243 29703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:44.809504 29703 hybrid_clock.cc:648] HybridClock initialized: now 1777492784809414 us; error 70 us; skew 500 ppm
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:44.812881 29703 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 19:59:44.814054 29703 webserver.cc:492] Webserver started at http://127.27.17.194:44317/ using document root <none> and password file <none>
I20260429 19:59:44.814646 29703 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:44.814697 29703 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:44.814862 29703 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:44.816763 29703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "e1cd42c809074e90a7fc0a3498629296"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "b4d7763349c2b55bea0d0ce0df3e1457"
server_key_iv: "08e64d6412d25aed5f098fdfde3e644b"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.817206 29703 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "e1cd42c809074e90a7fc0a3498629296"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "b4d7763349c2b55bea0d0ce0df3e1457"
server_key_iv: "08e64d6412d25aed5f098fdfde3e644b"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.820784 29703 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 19:59:44.823418 29718 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:44.824705 29703 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260429 19:59:44.825269 29703 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "e1cd42c809074e90a7fc0a3498629296"
format_stamp: "Formatted at 2026-04-29 19:59:44 on dist-test-slave-01nr"
server_key: "b4d7763349c2b55bea0d0ce0df3e1457"
server_key_iv: "08e64d6412d25aed5f098fdfde3e644b"
server_key_version: "encryptionkey@0"
I20260429 19:59:44.825475 29703 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:44.855706 29703 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:44.859062 29703 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:44.859288 29703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:44.861045 29703 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:44.861153 29703 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:44.861218 29703 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:44.861272 29703 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 19:59:44 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:44.873087 29703 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:39263
I20260429 19:59:44.873155 29833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:39263 every 8 connection(s)
I20260429 19:59:44.873904 29727 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.862106 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.27.17.194:45985)
0429 19:59:44.862527 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:44.862577 (+ 50us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:44.863424 (+ 847us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:44.864001 (+ 577us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:44.864010 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:44.864509 (+ 499us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:44.865071 (+ 562us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.865088 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.866019 (+ 931us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.866023 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:44.866524 (+ 501us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.866535 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.866753 (+ 218us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.867458 (+ 705us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:44.867478 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:44.869190 (+ 1712us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:44.871230 (+ 2040us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.871236 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.871249 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.871543 (+ 294us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.871828 (+ 285us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.871832 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.871851 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.871977 (+ 126us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.872362 (+ 385us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:44.872370 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:44.872620 (+ 250us) client_negotiation.cc:770] Sending connection context
0429 19:59:44.872948 (+ 328us) client_negotiation.cc:241] Negotiation successful
0429 19:59:44.873277 (+ 329us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":132,"threads_started":1}
I20260429 19:59:44.874214 29703 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 19:59:44.874591 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.861791 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:45985 (local address 127.27.17.254:35247)
0429 19:59:44.861953 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:44.861958 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:44.862699 (+ 741us) server_negotiation.cc:408] Connection header received
0429 19:59:44.863658 (+ 959us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:44.863662 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:44.863729 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:44.863848 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:44.865235 (+ 1387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.865805 (+ 570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.866729 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.867013 (+ 284us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.869372 (+ 2359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:44.869394 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:44.869397 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:44.869427 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:44.871099 (+ 1672us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.871658 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.871662 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.871664 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.871723 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.872075 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.872079 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.872081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.872248 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:44.872351 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:44.874267 (+ 1916us) server_negotiation.cc:300] Negotiation successful
0429 19:59:44.874408 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 19:59:44.876168 29834 heartbeater.cc:344] Connected to a master server at 127.27.17.254:35247
I20260429 19:59:44.876353 29834 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:44.876997 29834 heartbeater.cc:507] Master 127.27.17.254:35247 requested a full tablet report, sending...
I20260429 19:59:44.878115 29486 ts_manager.cc:194] Registered new tserver with Master: e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:44.878885 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 29703
I20260429 19:59:44.878995 29486 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:45985
I20260429 19:59:44.879040 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 19:59:44.879319 27719 external_mini_cluster.cc:1468] Setting key 9efd5c1963e89f71c02726caf5143e7d
I20260429 19:59:44.886488 29840 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.878097 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.27.17.194:42157)
0429 19:59:44.879456 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:44.879473 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:44.879627 (+ 154us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:44.879919 (+ 292us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:44.879923 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:44.880115 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:44.880334 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.880340 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.881346 (+ 1006us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.881353 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:44.881967 (+ 614us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:44.881979 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.882108 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.882757 (+ 649us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:44.882779 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:44.883132 (+ 353us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:44.885090 (+ 1958us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.885093 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.885095 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.885262 (+ 167us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.885558 (+ 296us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:44.885561 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:44.885562 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:44.885626 (+ 64us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:44.886083 (+ 457us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:44.886085 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:44.886154 (+ 69us) client_negotiation.cc:770] Sending connection context
0429 19:59:44.886233 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 19:59:44.886338 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1264,"thread_start_us":112,"threads_started":1}
I20260429 19:59:44.886683 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:44.878228 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:42157 (local address 127.27.17.254:35247)
0429 19:59:44.878381 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:44.878384 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:44.879545 (+ 1161us) server_negotiation.cc:408] Connection header received
0429 19:59:44.879754 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:44.879758 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:44.879817 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:44.879900 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:44.880489 (+ 589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.881208 (+ 719us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:44.882097 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:44.882308 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:44.883252 (+ 944us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:44.883273 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:44.883280 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:44.883313 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:44.884965 (+ 1652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.885377 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.885385 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.885389 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.885453 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:44.885718 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:44.885724 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:44.885729 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:44.885986 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:44.886105 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:44.886326 (+ 221us) server_negotiation.cc:300] Negotiation successful
0429 19:59:44.886470 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:44.940043 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35247
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:36753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:45.049016 29845 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:45.049258 29845 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:45.049326 29845 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:45.052780 29845 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:45.052860 29845 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:45.052917 29845 flags.cc:432] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260429 19:59:45.052932 29845 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 19:59:45.052975 29845 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 19:59:45.057770 29845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:36753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:35247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:45.059680 29845 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:45.060650 29845 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 19:59:45.067863 29845 server_base.cc:1061] running on GCE node
W20260429 19:59:45.067828 29851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:45.067808 29853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:45.067808 29850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:45.068570 29845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:45.069199 29845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:45.070390 29845 hybrid_clock.cc:648] HybridClock initialized: now 1777492785070365 us; error 39 us; skew 500 ppm
Apr 29 19:59:45 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492785, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:45.073601 29845 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 19:59:45.074803 29845 webserver.cc:492] Webserver started at http://127.27.17.195:42427/ using document root <none> and password file <none>
I20260429 19:59:45.075433 29845 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:45.075524 29845 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:45.075753 29845 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:45.077584 29845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "366cbb43e2504bc19f71a50964fdb6d4"
format_stamp: "Formatted at 2026-04-29 19:59:45 on dist-test-slave-01nr"
server_key: "3f25ce2db4dfbd97cdc933a3f109c4a0"
server_key_iv: "dcb510fdc68838e85e3752a339b47539"
server_key_version: "encryptionkey@0"
I20260429 19:59:45.078106 29845 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "366cbb43e2504bc19f71a50964fdb6d4"
format_stamp: "Formatted at 2026-04-29 19:59:45 on dist-test-slave-01nr"
server_key: "3f25ce2db4dfbd97cdc933a3f109c4a0"
server_key_iv: "dcb510fdc68838e85e3752a339b47539"
server_key_version: "encryptionkey@0"
I20260429 19:59:45.081832 29845 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260429 19:59:45.084347 29860 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:45.085508 29845 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:45.085669 29845 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "366cbb43e2504bc19f71a50964fdb6d4"
format_stamp: "Formatted at 2026-04-29 19:59:45 on dist-test-slave-01nr"
server_key: "3f25ce2db4dfbd97cdc933a3f109c4a0"
server_key_iv: "dcb510fdc68838e85e3752a339b47539"
server_key_version: "encryptionkey@0"
I20260429 19:59:45.085786 29845 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:45.106649 29845 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:45.109871 29845 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:45.110096 29845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:45.112017 29845 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:45.112104 29845 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:45.112185 29845 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:45.112229 29845 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 19:59:45 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492785, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:45.124738 29845 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:44807
I20260429 19:59:45.124817 29975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:44807 every 8 connection(s)
I20260429 19:59:45.125867 29845 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 19:59:45.126210 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.112512 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:59105 (local address 127.27.17.254:35247)
0429 19:59:45.112671 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.112676 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.113599 (+ 923us) server_negotiation.cc:408] Connection header received
0429 19:59:45.114464 (+ 865us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.114468 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.114546 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.114661 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:45.116418 (+ 1757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.117299 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.118341 (+ 1042us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.118583 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.121634 (+ 3051us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:45.121666 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:45.121669 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:45.121706 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:45.123945 (+ 2239us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.124606 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.124609 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.124611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.124667 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.125035 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.125039 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.125041 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.125259 (+ 218us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:45.125411 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.125910 (+ 499us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.126044 (+ 134us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 19:59:45.126837 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 29845
I20260429 19:59:45.126976 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 19:59:45.127214 27719 external_mini_cluster.cc:1468] Setting key 150fe4079ef597bde7e31989db23ee8a
I20260429 19:59:45.127004 29869 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.112876 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.27.17.195:59105)
0429 19:59:45.113325 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.113368 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.114230 (+ 862us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.114873 (+ 643us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.114887 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.115429 (+ 542us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:45.116196 (+ 767us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.116215 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.117484 (+ 1269us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.117490 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.118168 (+ 678us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.118182 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.118425 (+ 243us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.119174 (+ 749us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:45.119204 (+ 30us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:45.121353 (+ 2149us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:45.124101 (+ 2748us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:45.124109 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:45.124123 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:45.124439 (+ 316us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:45.124800 (+ 361us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:45.124803 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:45.124805 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:45.124941 (+ 136us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:45.125409 (+ 468us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:45.125416 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:45.125659 (+ 243us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.125871 (+ 212us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.126121 (+ 250us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":129,"threads_started":1}
I20260429 19:59:45.129366 29976 heartbeater.cc:344] Connected to a master server at 127.27.17.254:35247
I20260429 19:59:45.129547 29976 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:45.130121 29976 heartbeater.cc:507] Master 127.27.17.254:35247 requested a full tablet report, sending...
I20260429 19:59:45.131227 29487 ts_manager.cc:194] Registered new tserver with Master: 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807)
I20260429 19:59:45.131953 29487 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:59105
I20260429 19:59:45.138226 29982 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.131570 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.27.17.195:37353)
0429 19:59:45.131959 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.131974 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.132124 (+ 150us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.132471 (+ 347us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.132475 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.132680 (+ 205us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:45.132886 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.132891 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.133892 (+ 1001us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.133895 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.134260 (+ 365us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.134266 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.134366 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.134950 (+ 584us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:45.134971 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:45.135254 (+ 283us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:45.136888 (+ 1634us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:45.136892 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:45.136898 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:45.137102 (+ 204us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:45.137387 (+ 285us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:45.137390 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:45.137392 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:45.137442 (+ 50us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:45.137826 (+ 384us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:45.137829 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:45.137897 (+ 68us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.137983 (+ 86us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.138083 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
I20260429 19:59:45.138314 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.131802 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:37353 (local address 127.27.17.254:35247)
0429 19:59:45.131908 (+ 106us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.131913 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.132055 (+ 142us) server_negotiation.cc:408] Connection header received
0429 19:59:45.132294 (+ 239us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.132297 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.132356 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.132417 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:45.133009 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.133770 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.134385 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.134553 (+ 168us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.135400 (+ 847us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:45.135416 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:45.135418 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:45.135442 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:45.136762 (+ 1320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.137215 (+ 453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.137218 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.137220 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.137275 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.137590 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.137593 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.137594 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.137738 (+ 144us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:45.137824 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.138077 (+ 253us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.138185 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260429 19:59:45.141644 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 19:59:45 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 19:59:45 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:45.166946 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.158503 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.27.17.254:35247)
0429 19:59:45.158666 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.158670 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.158683 (+ 13us) server_negotiation.cc:408] Connection header received
0429 19:59:45.158724 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.158726 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.158771 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.158848 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:45.159861 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.160370 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.161059 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.161245 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.163545 (+ 2300us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:45.163568 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:45.163571 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:45.163605 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:45.165130 (+ 1525us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.165580 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.165584 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.165585 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.165643 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.165938 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.165941 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.165942 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.166106 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:45.166214 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.166540 (+ 326us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.166726 (+ 186us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260429 19:59:45.169998 29487 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 19:59:45.172595 29487 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:45.185863 29995 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.181362 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.0.0.1:45868)
0429 19:59:45.182196 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.182237 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.182374 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.183123 (+ 749us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.183128 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.183151 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.183447 (+ 296us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.183457 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.184606 (+ 1149us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.184618 (+ 12us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.185436 (+ 818us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.185445 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.185548 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.185619 (+ 71us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.185690 (+ 71us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.185751 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":99,"threads_started":1}
I20260429 19:59:45.186705 29993 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.181501 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:39263 (local address 127.0.0.1:48076)
0429 19:59:45.182196 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.182237 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.182374 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.183475 (+ 1101us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.183481 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.183505 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.183837 (+ 332us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.183845 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.185325 (+ 1480us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.185330 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.186316 (+ 986us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.186324 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.186455 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.186470 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.186522 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.186577 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":112,"threads_started":1}
I20260429 19:59:45.186954 29994 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.181493 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.27.17.193:44421)
0429 19:59:45.181811 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.181818 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.182498 (+ 680us) server_negotiation.cc:408] Connection header received
0429 19:59:45.182589 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.182595 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.182784 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.182952 (+ 168us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.183590 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.184487 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.186103 (+ 1616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.186517 (+ 414us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.186618 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.186682 (+ 64us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.186749 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":103,"threads_started":1}
I20260429 19:59:45.187582 29997 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.182190 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:44807 (local address 127.0.0.1:48510)
0429 19:59:45.182669 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.182682 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.182754 (+ 72us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.183609 (+ 855us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.183612 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.183635 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.183949 (+ 314us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.183956 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.185350 (+ 1394us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.185353 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.187222 (+ 1869us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.187232 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.187360 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.187373 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.187419 (+ 46us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.187470 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":76,"threads_started":1}
I20260429 19:59:45.188249 29998 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.182584 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.27.17.195:44807)
0429 19:59:45.183077 (+ 493us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.183083 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.183113 (+ 30us) server_negotiation.cc:408] Connection header received
0429 19:59:45.183207 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.183214 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.183414 (+ 200us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.183566 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.184052 (+ 486us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.185221 (+ 1169us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.187384 (+ 2163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.187881 (+ 497us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.187962 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.188016 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.188113 (+ 97us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":102,"threads_started":1}
I20260429 19:59:45.188263 29996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.182030 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.27.17.194:39263)
0429 19:59:45.183033 (+ 1003us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.183039 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.183063 (+ 24us) server_negotiation.cc:408] Connection header received
0429 19:59:45.183144 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.183149 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.183296 (+ 147us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.183424 (+ 128us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.183963 (+ 539us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.185176 (+ 1213us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.186457 (+ 1281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.187864 (+ 1407us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.187997 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.188060 (+ 63us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.188143 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":110,"threads_started":1}
I20260429 19:59:45.190147 29626 tablet_service.cc:1511] Processing CreateTablet for tablet 73242f7b101a40e09b500d0b94a8ec34 (DEFAULT_TABLE table=test-table [id=0b9a71c4a9f84ac58f70e735aff2fabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:45.191296 29626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73242f7b101a40e09b500d0b94a8ec34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.191397 29768 tablet_service.cc:1511] Processing CreateTablet for tablet 73242f7b101a40e09b500d0b94a8ec34 (DEFAULT_TABLE table=test-table [id=0b9a71c4a9f84ac58f70e735aff2fabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:45.191491 29910 tablet_service.cc:1511] Processing CreateTablet for tablet 73242f7b101a40e09b500d0b94a8ec34 (DEFAULT_TABLE table=test-table [id=0b9a71c4a9f84ac58f70e735aff2fabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:45.192307 29768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73242f7b101a40e09b500d0b94a8ec34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.192351 29910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73242f7b101a40e09b500d0b94a8ec34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.197381 29999 tablet_bootstrap.cc:492] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Bootstrap starting.
I20260429 19:59:45.199097 30001 tablet_bootstrap.cc:492] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Bootstrap starting.
I20260429 19:59:45.199143 30000 tablet_bootstrap.cc:492] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Bootstrap starting.
I20260429 19:59:45.199944 29999 tablet_bootstrap.cc:654] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.200911 30000 tablet_bootstrap.cc:654] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.201296 30001 tablet_bootstrap.cc:654] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.201274 29999 log.cc:826] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:45.201788 30000 log.cc:826] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:45.202764 30001 log.cc:826] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:45.203778 29999 tablet_bootstrap.cc:492] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: No bootstrap required, opened a new log
I20260429 19:59:45.203825 30000 tablet_bootstrap.cc:492] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: No bootstrap required, opened a new log
I20260429 19:59:45.203987 29999 ts_tablet_manager.cc:1403] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260429 19:59:45.204006 30000 ts_tablet_manager.cc:1403] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260429 19:59:45.204473 30001 tablet_bootstrap.cc:492] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: No bootstrap required, opened a new log
I20260429 19:59:45.204648 30001 ts_tablet_manager.cc:1403] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 19:59:45.206825 29999 raft_consensus.cc:359] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.207126 29999 raft_consensus.cc:385] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.207191 29999 raft_consensus.cc:740] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cd42c809074e90a7fc0a3498629296, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.207240 30000 raft_consensus.cc:359] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.207441 30000 raft_consensus.cc:385] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.207490 30000 raft_consensus.cc:740] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa60c1192024a6ca11d0c2561fc4619, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.207624 30001 raft_consensus.cc:359] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.207674 29999 consensus_queue.cc:260] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.207800 30001 raft_consensus.cc:385] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.207850 30001 raft_consensus.cc:740] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.207916 30000 consensus_queue.cc:260] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.208218 30001 consensus_queue.cc:260] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.208642 29692 heartbeater.cc:499] Master 127.27.17.254:35247 was elected leader, sending a full tablet report...
I20260429 19:59:45.208766 29999 ts_tablet_manager.cc:1434] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 19:59:45.209019 30001 ts_tablet_manager.cc:1434] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260429 19:59:45.209112 29834 heartbeater.cc:499] Master 127.27.17.254:35247 was elected leader, sending a full tablet report...
I20260429 19:59:45.209358 29976 heartbeater.cc:499] Master 127.27.17.254:35247 was elected leader, sending a full tablet report...
I20260429 19:59:45.209975 30000 ts_tablet_manager.cc:1434] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260429 19:59:45.354364 29693 tablet.cc:2404] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 19:59:45.362716 30007 raft_consensus.cc:493] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:45.362973 30007 raft_consensus.cc:515] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.364148 30007 leader_election.cc:290] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:45.367511 29869 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.364490 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:39263 (local address 127.27.17.195:34691)
0429 19:59:45.364629 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.364646 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.364753 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.365137 (+ 384us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.365141 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.365169 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.365449 (+ 280us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.365470 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.366401 (+ 931us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.366406 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.367131 (+ 725us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.367140 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.367238 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.367253 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.367318 (+ 65us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.367378 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260429 19:59:45.367998 30008 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.364490 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.27.17.195:57227)
0429 19:59:45.365017 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.365032 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.365126 (+ 94us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.365404 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.365406 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.365422 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.365760 (+ 338us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.365768 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.366607 (+ 839us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.366610 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.367230 (+ 620us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.367239 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.367388 (+ 149us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.367403 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.367453 (+ 50us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.367502 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":135,"thread_start_us":157,"threads_started":1}
I20260429 19:59:45.368090 29996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.364684 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:34691 (local address 127.27.17.194:39263)
0429 19:59:45.364802 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.364806 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.364840 (+ 34us) server_negotiation.cc:408] Connection header received
0429 19:59:45.364899 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.364903 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.364958 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.365063 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.365618 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.366278 (+ 660us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.367266 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.367817 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.367855 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.367914 (+ 59us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.367966 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 19:59:45.368109 29994 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.364572 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:57227 (local address 127.27.17.193:44421)
0429 19:59:45.364718 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.364722 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.365136 (+ 414us) server_negotiation.cc:408] Connection header received
0429 19:59:45.365206 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.365210 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.365283 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.365388 (+ 105us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.365881 (+ 493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.366489 (+ 608us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.367372 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.367866 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.367898 (+ 32us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.367944 (+ 46us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.367987 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260429 19:59:45.368665 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73242f7b101a40e09b500d0b94a8ec34" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296" is_pre_election: true
I20260429 19:59:45.368757 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73242f7b101a40e09b500d0b94a8ec34" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619" is_pre_election: true
I20260429 19:59:45.368985 29788 raft_consensus.cc:2468] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 0.
I20260429 19:59:45.369032 29646 raft_consensus.cc:2468] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 0.
I20260429 19:59:45.369489 29861 leader_election.cc:304] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366cbb43e2504bc19f71a50964fdb6d4, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:45.374208 30007 raft_consensus.cc:2804] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:45.374341 30007 raft_consensus.cc:493] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:45.374402 30007 raft_consensus.cc:3060] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Advancing to term 1
W20260429 19:59:45.375640 29835 tablet.cc:2404] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 19:59:45.376786 30007 raft_consensus.cc:515] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
W20260429 19:59:45.377240 29977 tablet.cc:2404] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 19:59:45.377476 30007 leader_election.cc:290] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:45.377863 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73242f7b101a40e09b500d0b94a8ec34" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619"
I20260429 19:59:45.377971 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73242f7b101a40e09b500d0b94a8ec34" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296"
I20260429 19:59:45.378048 29646 raft_consensus.cc:3060] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.378136 29788 raft_consensus.cc:3060] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.379428 29788 raft_consensus.cc:2468] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 1.
I20260429 19:59:45.379505 29646 raft_consensus.cc:2468] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 1.
I20260429 19:59:45.379782 29861 leader_election.cc:304] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366cbb43e2504bc19f71a50964fdb6d4, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:45.379999 30007 raft_consensus.cc:2804] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:45.380194 30007 raft_consensus.cc:697] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Running, Role: LEADER
I20260429 19:59:45.380506 30007 consensus_queue.cc:237] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.383776 29486 catalog_manager.cc:5671] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195). New cstate: current_term: 1 leader_uuid: "366cbb43e2504bc19f71a50964fdb6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } health_report { overall_health: HEALTHY } } }
Apr 29 19:59:45 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492784, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:45.457135 29536 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.448953 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.27.17.254:35247)
0429 19:59:45.449101 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.449105 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.449345 (+ 240us) server_negotiation.cc:408] Connection header received
0429 19:59:45.449444 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.449447 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.449492 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.449578 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:45.450443 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.450989 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.451647 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.451779 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.453948 (+ 2169us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:45.453970 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:45.453972 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:45.453997 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:45.455532 (+ 1535us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.456088 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.456090 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.456092 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.456133 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:45.456380 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:45.456382 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:45.456383 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:45.456526 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:45.456619 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.456880 (+ 261us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.457005 (+ 125us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":58}
I20260429 19:59:45.457136 30016 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.448912 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35247 (local address 127.0.0.1:46330)
0429 19:59:45.449209 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.449226 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.449359 (+ 133us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.449635 (+ 276us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.449638 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.450072 (+ 434us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:45.450278 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.450285 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.451113 (+ 828us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.451116 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.451531 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.451538 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.451638 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.452189 (+ 551us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:45.452202 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:45.453819 (+ 1617us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:45.455740 (+ 1921us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:45.455748 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:45.455752 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:45.455990 (+ 238us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:45.456212 (+ 222us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:45.456215 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:45.456216 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:45.456296 (+ 80us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:45.456635 (+ 339us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:45.456639 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:45.456734 (+ 95us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.456825 (+ 91us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.456946 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":137,"threads_started":1}
I20260429 19:59:45.461743 29486 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:46330:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 19:59:45.462641 29486 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:45.469677 29768 tablet_service.cc:1511] Processing CreateTablet for tablet 4a591624504d4872b27e1a9ea48b6134 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a869fc3c531c4a88bc82d9136f1c1ebb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 19:59:45.469429 29626 tablet_service.cc:1511] Processing CreateTablet for tablet 4a591624504d4872b27e1a9ea48b6134 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a869fc3c531c4a88bc82d9136f1c1ebb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 19:59:45.469946 29910 tablet_service.cc:1511] Processing CreateTablet for tablet 4a591624504d4872b27e1a9ea48b6134 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a869fc3c531c4a88bc82d9136f1c1ebb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 19:59:45.470337 29768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a591624504d4872b27e1a9ea48b6134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.470333 29626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a591624504d4872b27e1a9ea48b6134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.470440 29910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a591624504d4872b27e1a9ea48b6134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.473057 29999 tablet_bootstrap.cc:492] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: Bootstrap starting.
I20260429 19:59:45.473134 30000 tablet_bootstrap.cc:492] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: Bootstrap starting.
I20260429 19:59:45.473134 30001 tablet_bootstrap.cc:492] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: Bootstrap starting.
I20260429 19:59:45.474145 29999 tablet_bootstrap.cc:654] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.474195 30001 tablet_bootstrap.cc:654] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.474195 30000 tablet_bootstrap.cc:654] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.475690 30001 tablet_bootstrap.cc:492] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: No bootstrap required, opened a new log
I20260429 19:59:45.475721 30000 tablet_bootstrap.cc:492] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: No bootstrap required, opened a new log
I20260429 19:59:45.475795 30001 ts_tablet_manager.cc:1403] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 19:59:45.475834 30000 ts_tablet_manager.cc:1403] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260429 19:59:45.475934 29999 tablet_bootstrap.cc:492] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: No bootstrap required, opened a new log
I20260429 19:59:45.476027 29999 ts_tablet_manager.cc:1403] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 19:59:45.476383 30001 raft_consensus.cc:359] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.476419 30000 raft_consensus.cc:359] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.476505 30001 raft_consensus.cc:385] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.476544 30001 raft_consensus.cc:740] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.476541 30000 raft_consensus.cc:385] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.476501 29999 raft_consensus.cc:359] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.476580 30000 raft_consensus.cc:740] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa60c1192024a6ca11d0c2561fc4619, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.476600 29999 raft_consensus.cc:385] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.476632 29999 raft_consensus.cc:740] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cd42c809074e90a7fc0a3498629296, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.476737 30001 consensus_queue.cc:260] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.476786 29999 consensus_queue.cc:260] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.476816 30000 consensus_queue.cc:260] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.477010 30007 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.477072 29999 ts_tablet_manager.cc:1434] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:45.477135 30007 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.477337 30005 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.477448 30005 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.477782 30001 ts_tablet_manager.cc:1434] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260429 19:59:45.478137 30000 ts_tablet_manager.cc:1434] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:45.478116 30006 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.478229 30006 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.579346 30006 raft_consensus.cc:493] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:45.579564 30006 raft_consensus.cc:515] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.580786 30006 leader_election.cc:290] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263), 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807)
I20260429 19:59:45.584044 30020 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.580983 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:39263 (local address 127.27.17.193:46789)
0429 19:59:45.581298 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.581314 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.581492 (+ 178us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.581795 (+ 303us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.581798 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.581817 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.582061 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.582067 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.582785 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.582789 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.583434 (+ 645us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.583441 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.583587 (+ 146us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.583606 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.583660 (+ 54us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.583877 (+ 217us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":137,"threads_started":1}
I20260429 19:59:45.584314 29996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.581130 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:46789 (local address 127.27.17.194:39263)
0429 19:59:45.581294 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.581329 (+ 35us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.581393 (+ 64us) server_negotiation.cc:408] Connection header received
0429 19:59:45.581622 (+ 229us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.581626 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.581681 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.581770 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.582177 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.582667 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.583569 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.584025 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.584066 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.584125 (+ 59us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.584178 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 19:59:45.584410 29998 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.581385 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:52723 (local address 127.27.17.195:44807)
0429 19:59:45.581545 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.581572 (+ 27us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.581588 (+ 16us) server_negotiation.cc:408] Connection header received
0429 19:59:45.581744 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.581749 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.581796 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.581881 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.582293 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.582802 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.583640 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.584148 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.584192 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.584244 (+ 52us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.584289 (+ 45us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 19:59:45.584437 30021 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.581127 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:44807 (local address 127.27.17.193:52723)
0429 19:59:45.581504 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.581518 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.581621 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.581893 (+ 272us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.581896 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.581914 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.582174 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.582179 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.582944 (+ 765us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.582950 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.583517 (+ 567us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.583524 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.583672 (+ 148us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.583688 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.583741 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.583790 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260429 19:59:45.584785 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a591624504d4872b27e1a9ea48b6134" candidate_uuid: "5fa60c1192024a6ca11d0c2561fc4619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296" is_pre_election: true
I20260429 19:59:45.584977 29788 raft_consensus.cc:2468] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa60c1192024a6ca11d0c2561fc4619 in term 0.
I20260429 19:59:45.585144 29930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a591624504d4872b27e1a9ea48b6134" candidate_uuid: "5fa60c1192024a6ca11d0c2561fc4619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366cbb43e2504bc19f71a50964fdb6d4" is_pre_election: true
I20260429 19:59:45.585361 29930 raft_consensus.cc:2468] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fa60c1192024a6ca11d0c2561fc4619 in term 0.
I20260429 19:59:45.585361 29577 leader_election.cc:304] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa60c1192024a6ca11d0c2561fc4619, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:45.585638 30006 raft_consensus.cc:2804] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:45.585705 30006 raft_consensus.cc:493] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:45.585765 30006 raft_consensus.cc:3060] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.586688 30006 raft_consensus.cc:515] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.587157 30006 leader_election.cc:290] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [CANDIDATE]: Term 1 election: Requested vote from peers e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263), 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807)
I20260429 19:59:45.587601 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a591624504d4872b27e1a9ea48b6134" candidate_uuid: "5fa60c1192024a6ca11d0c2561fc4619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296"
I20260429 19:59:45.587733 29788 raft_consensus.cc:3060] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.587827 29930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a591624504d4872b27e1a9ea48b6134" candidate_uuid: "5fa60c1192024a6ca11d0c2561fc4619" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366cbb43e2504bc19f71a50964fdb6d4"
I20260429 19:59:45.587996 29930 raft_consensus.cc:3060] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.588682 29788 raft_consensus.cc:2468] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa60c1192024a6ca11d0c2561fc4619 in term 1.
I20260429 19:59:45.588979 29577 leader_election.cc:304] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa60c1192024a6ca11d0c2561fc4619, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:45.589143 30006 raft_consensus.cc:2804] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:45.589133 29930 raft_consensus.cc:2468] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fa60c1192024a6ca11d0c2561fc4619 in term 1.
I20260429 19:59:45.589411 30006 raft_consensus.cc:697] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 LEADER]: Becoming Leader. State: Replica: 5fa60c1192024a6ca11d0c2561fc4619, State: Running, Role: LEADER
I20260429 19:59:45.589761 30006 consensus_queue.cc:237] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.591939 30022 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: TxnStatusTablet state changed. Reason: New leader 5fa60c1192024a6ca11d0c2561fc4619. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.592052 30022 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: This TxnStatusTablet replica's current role is: LEADER
I20260429 19:59:45.592342 30024 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 19:59:45.592536 30024 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 19:59:45.592965 29486 catalog_manager.cc:5671] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193). New cstate: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } health_report { overall_health: UNKNOWN } } }
I20260429 19:59:45.634110 29994 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.631086 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.27.17.193:44421)
0429 19:59:45.631235 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.631239 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.631254 (+ 15us) server_negotiation.cc:408] Connection header received
0429 19:59:45.631428 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.631431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.631485 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.631584 (+ 99us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:45.631919 (+ 335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.632394 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.633061 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.633275 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.633333 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.633696 (+ 363us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.633813 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.633918 (+ 105us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.633965 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 19:59:45.634110 30016 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.630979 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.0.0.1:59126)
0429 19:59:45.631116 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.631140 (+ 24us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.631258 (+ 118us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.631592 (+ 334us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.631595 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.631603 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:45.631810 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.631815 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.632530 (+ 715us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.632533 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.632949 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.632955 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.633093 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.633137 (+ 44us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.633798 (+ 661us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.633810 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.633908 (+ 98us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.633981 (+ 73us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260429 19:59:45.639609 29788 raft_consensus.cc:1275] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Refusing update from remote peer 5fa60c1192024a6ca11d0c2561fc4619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:45.639609 29930 raft_consensus.cc:1275] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Refusing update from remote peer 5fa60c1192024a6ca11d0c2561fc4619: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:45.640291 30006 consensus_queue.cc:1048] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:45.640498 30022 consensus_queue.cc:1048] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:45.643853 30007 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: TxnStatusTablet state changed. Reason: New leader 5fa60c1192024a6ca11d0c2561fc4619. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.643978 30007 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.644337 30005 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: TxnStatusTablet state changed. Reason: New leader 5fa60c1192024a6ca11d0c2561fc4619. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.644438 30005 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.648291 30028 mvcc.cc:204] Tried to move back new op lower bound from 7280610449972895744 to 7280610449778139136. Current Snapshot: MvccSnapshot[applied={T|T < 7280610449972895744}]
I20260429 19:59:45.648310 30006 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.648427 30006 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: This TxnStatusTablet replica's current role is: LEADER
I20260429 19:59:45.648542 30006 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.648600 30006 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 5fa60c1192024a6ca11d0c2561fc4619: This TxnStatusTablet replica's current role is: LEADER
I20260429 19:59:45.648571 30007 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.648689 30007 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P 366cbb43e2504bc19f71a50964fdb6d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.648685 30027 mvcc.cc:204] Tried to move back new op lower bound from 7280610449972895744 to 7280610449778139136. Current Snapshot: MvccSnapshot[applied={T|T < 7280610449972895744}]
I20260429 19:59:45.649585 30005 tablet_replica.cc:442] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fa60c1192024a6ca11d0c2561fc4619" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } } }
I20260429 19:59:45.649695 30005 tablet_replica.cc:445] T 4a591624504d4872b27e1a9ea48b6134 P e1cd42c809074e90a7fc0a3498629296: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 19:59:45.650264 30029 mvcc.cc:204] Tried to move back new op lower bound from 7280610449972895744 to 7280610449778139136. Current Snapshot: MvccSnapshot[applied={T|T < 7280610449972895744}]
I20260429 19:59:45.664793 29998 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.660598 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.27.17.195:44807)
0429 19:59:45.660755 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.660759 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.660776 (+ 17us) server_negotiation.cc:408] Connection header received
0429 19:59:45.660888 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.660891 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.660965 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.661067 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:45.661459 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.662331 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.663484 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.663698 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.663770 (+ 72us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.664278 (+ 508us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.664446 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.664567 (+ 121us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.664630 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 19:59:45.676780 29994 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.673395 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:55199 (local address 127.27.17.193:44421)
0429 19:59:45.673562 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.673567 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.673712 (+ 145us) server_negotiation.cc:408] Connection header received
0429 19:59:45.673834 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.673838 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.673898 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.673984 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:45.674394 (+ 410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.675155 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.675960 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.676154 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.676198 (+ 44us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.676298 (+ 100us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.676378 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.676568 (+ 190us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.676616 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 19:59:45.676834 30039 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.673235 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.27.17.195:55199)
0429 19:59:45.673624 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.673637 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.673733 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.674002 (+ 269us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.674005 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.674012 (+ 7us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:45.674251 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.674255 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.675317 (+ 1062us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.675321 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.675836 (+ 515us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.675845 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.675954 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.675991 (+ 37us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.676431 (+ 440us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.676438 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.676517 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.676630 (+ 113us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":143,"threads_started":1}
I20260429 19:59:45.683949 29788 raft_consensus.cc:1275] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Refusing update from remote peer 366cbb43e2504bc19f71a50964fdb6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:45.683982 29646 raft_consensus.cc:1275] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Refusing update from remote peer 366cbb43e2504bc19f71a50964fdb6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:45.684427 30007 consensus_queue.cc:1048] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:45.684594 30009 consensus_queue.cc:1048] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:45.710687 29998 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.707444 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:57517 (local address 127.27.17.195:44807)
0429 19:59:45.707590 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.707594 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.707873 (+ 279us) server_negotiation.cc:408] Connection header received
0429 19:59:45.708038 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.708042 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.708101 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.708173 (+ 72us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:45.708605 (+ 432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.709214 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.709970 (+ 756us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.710171 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.710209 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.710278 (+ 69us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.710375 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.710506 (+ 131us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.710545 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 19:59:45.710721 30044 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.707420 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:44807 (local address 127.27.17.193:57517)
0429 19:59:45.707772 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.707785 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.707913 (+ 128us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.708184 (+ 271us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.708187 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.708196 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:45.708482 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.708488 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.709330 (+ 842us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.709338 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.709849 (+ 511us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.709856 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.709984 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.710031 (+ 47us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.710398 (+ 367us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.710410 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.710492 (+ 82us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.710583 (+ 91us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":124,"threads_started":1}
I20260429 19:59:45.739940 29486 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:45.740140 29486 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:45.743111 29486 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Sending DeleteTablet for 3 replicas of tablet 73242f7b101a40e09b500d0b94a8ec34
I20260429 19:59:45.743991 29626 tablet_service.cc:1558] Processing DeleteTablet for tablet 73242f7b101a40e09b500d0b94a8ec34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:45 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:45868
I20260429 19:59:45.744115 29768 tablet_service.cc:1558] Processing DeleteTablet for tablet 73242f7b101a40e09b500d0b94a8ec34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:45 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48076
I20260429 19:59:45.744827 29910 tablet_service.cc:1558] Processing DeleteTablet for tablet 73242f7b101a40e09b500d0b94a8ec34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:45 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48510
I20260429 19:59:45.745409 29486 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 19:59:45.745815 30049 tablet_replica.cc:333] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: stopping tablet replica
W20260429 19:59:45.745901 29486 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:45.746260 30049 raft_consensus.cc:2243] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 LEADER]: Raft consensus shutting down.
I20260429 19:59:45.746819 30049 raft_consensus.cc:2272] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:45.747141 30047 tablet_replica.cc:333] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: stopping tablet replica
I20260429 19:59:45.747785 30048 tablet_replica.cc:333] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: stopping tablet replica
I20260429 19:59:45.749316 30049 ts_tablet_manager.cc:1916] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:45.749545 30047 raft_consensus.cc:2243] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:45.749778 30047 raft_consensus.cc:2272] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:45.753060 29910 tablet_service.cc:1511] Processing CreateTablet for tablet 4b61d78ffbeb46958a7de3a4abffb632 (DEFAULT_TABLE table=test-table [id=99369559b5c64ac080f919204edda338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:45.753103 29626 tablet_service.cc:1511] Processing CreateTablet for tablet 4b61d78ffbeb46958a7de3a4abffb632 (DEFAULT_TABLE table=test-table [id=99369559b5c64ac080f919204edda338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:45.753340 29910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b61d78ffbeb46958a7de3a4abffb632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.753404 29626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b61d78ffbeb46958a7de3a4abffb632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.755635 30001 tablet_bootstrap.cc:492] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Bootstrap starting.
I20260429 19:59:45.756685 30001 tablet_bootstrap.cc:654] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.756711 29768 tablet_service.cc:1511] Processing CreateTablet for tablet 4b61d78ffbeb46958a7de3a4abffb632 (DEFAULT_TABLE table=test-table [id=99369559b5c64ac080f919204edda338]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:45.757004 29768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b61d78ffbeb46958a7de3a4abffb632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:45.757328 30000 tablet_bootstrap.cc:492] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Bootstrap starting.
I20260429 19:59:45.758311 30000 tablet_bootstrap.cc:654] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.759665 30049 ts_tablet_manager.cc:1929] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:45.759837 30049 log.cc:1199] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/73242f7b101a40e09b500d0b94a8ec34
I20260429 19:59:45.761008 30047 ts_tablet_manager.cc:1916] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:45.761092 29999 tablet_bootstrap.cc:492] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Bootstrap starting.
I20260429 19:59:45.762461 29999 tablet_bootstrap.cc:654] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:45.764472 30047 ts_tablet_manager.cc:1929] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:45.764734 30048 raft_consensus.cc:2243] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:45.765002 30049 ts_tablet_manager.cc:1950] T 73242f7b101a40e09b500d0b94a8ec34 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting consensus metadata
I20260429 19:59:45.765027 30048 raft_consensus.cc:2272] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:45.765826 30047 log.cc:1199] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/73242f7b101a40e09b500d0b94a8ec34
I20260429 19:59:45.766393 30047 ts_tablet_manager.cc:1950] T 73242f7b101a40e09b500d0b94a8ec34 P 5fa60c1192024a6ca11d0c2561fc4619: Deleting consensus metadata
I20260429 19:59:45.767304 29472 catalog_manager.cc:5002] TS 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807): tablet 73242f7b101a40e09b500d0b94a8ec34 (table test-table [id=0b9a71c4a9f84ac58f70e735aff2fabc]) successfully deleted
I20260429 19:59:45.767676 30048 ts_tablet_manager.cc:1916] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:45.768862 29472 catalog_manager.cc:5002] TS 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421): tablet 73242f7b101a40e09b500d0b94a8ec34 (table test-table [id=0b9a71c4a9f84ac58f70e735aff2fabc]) successfully deleted
I20260429 19:59:45.770283 30000 tablet_bootstrap.cc:492] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: No bootstrap required, opened a new log
I20260429 19:59:45.770387 30000 ts_tablet_manager.cc:1403] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20260429 19:59:45.770975 30000 raft_consensus.cc:359] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.771103 30000 raft_consensus.cc:385] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.771138 30000 raft_consensus.cc:740] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa60c1192024a6ca11d0c2561fc4619, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.771275 30000 consensus_queue.cc:260] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.771590 30000 ts_tablet_manager.cc:1434] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:45.771999 30048 ts_tablet_manager.cc:1929] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:45.772091 30048 log.cc:1199] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/73242f7b101a40e09b500d0b94a8ec34
I20260429 19:59:45.772392 30048 ts_tablet_manager.cc:1950] T 73242f7b101a40e09b500d0b94a8ec34 P e1cd42c809074e90a7fc0a3498629296: Deleting consensus metadata
I20260429 19:59:45.773705 29471 catalog_manager.cc:5002] TS e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263): tablet 73242f7b101a40e09b500d0b94a8ec34 (table test-table [id=0b9a71c4a9f84ac58f70e735aff2fabc]) successfully deleted
I20260429 19:59:45.774750 30001 tablet_bootstrap.cc:492] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: No bootstrap required, opened a new log
I20260429 19:59:45.774881 30001 ts_tablet_manager.cc:1403] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20260429 19:59:45.775332 29999 tablet_bootstrap.cc:492] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: No bootstrap required, opened a new log
I20260429 19:59:45.775429 29999 ts_tablet_manager.cc:1403] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260429 19:59:45.775482 30001 raft_consensus.cc:359] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.775610 30001 raft_consensus.cc:385] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.775648 30001 raft_consensus.cc:740] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.775787 30001 consensus_queue.cc:260] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.775913 29999 raft_consensus.cc:359] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.776021 29999 raft_consensus.cc:385] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:45.776058 29999 raft_consensus.cc:740] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cd42c809074e90a7fc0a3498629296, State: Initialized, Role: FOLLOWER
I20260429 19:59:45.776087 30001 ts_tablet_manager.cc:1434] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 19:59:45.776208 29999 consensus_queue.cc:260] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.776492 29999 ts_tablet_manager.cc:1434] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 19:59:45.909272 30005 raft_consensus.cc:493] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:45.909505 30005 raft_consensus.cc:515] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.910768 30005 leader_election.cc:290] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807)
I20260429 19:59:45.915004 30054 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.911489 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:44807 (local address 127.27.17.194:37935)
0429 19:59:45.911844 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.911859 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.911972 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.912274 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.912278 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.912302 (+ 24us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.912599 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.912609 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.913563 (+ 954us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.913565 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.914540 (+ 975us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.914549 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.914673 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.914693 (+ 20us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.914746 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.914805 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":93,"threads_started":1}
I20260429 19:59:45.915002 30053 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.911116 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.27.17.194:48161)
0429 19:59:45.911453 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.911469 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.911650 (+ 181us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.912154 (+ 504us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.912157 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.912179 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 19:59:45.912469 (+ 290us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.912478 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.913540 (+ 1062us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.913544 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.914540 (+ 996us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.914549 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.914673 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.914694 (+ 21us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.914746 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.914805 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":79,"threads_started":1}
I20260429 19:59:45.915789 29994 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.911299 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:48161 (local address 127.27.17.193:44421)
0429 19:59:45.911452 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.911467 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.911555 (+ 88us) server_negotiation.cc:408] Connection header received
0429 19:59:45.911807 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.911811 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.911897 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.911979 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.912597 (+ 618us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.913384 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.914985 (+ 1601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.915475 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.915533 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.915591 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.915657 (+ 66us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 19:59:45.916136 29998 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.911697 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:37935 (local address 127.27.17.195:44807)
0429 19:59:45.911834 (+ 137us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.911838 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.911946 (+ 108us) server_negotiation.cc:408] Connection header received
0429 19:59:45.912083 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.912090 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.912146 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.912229 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 19:59:45.912742 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.913448 (+ 706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.914686 (+ 1238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.915822 (+ 1136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.915880 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.915956 (+ 76us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.916008 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 19:59:45.916294 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b61d78ffbeb46958a7de3a4abffb632" candidate_uuid: "e1cd42c809074e90a7fc0a3498629296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619" is_pre_election: true
I20260429 19:59:45.916472 29646 raft_consensus.cc:2468] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cd42c809074e90a7fc0a3498629296 in term 0.
I20260429 19:59:45.916589 29930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b61d78ffbeb46958a7de3a4abffb632" candidate_uuid: "e1cd42c809074e90a7fc0a3498629296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366cbb43e2504bc19f71a50964fdb6d4" is_pre_election: true
I20260429 19:59:45.916749 29930 raft_consensus.cc:2468] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1cd42c809074e90a7fc0a3498629296 in term 0.
I20260429 19:59:45.917146 29720 leader_election.cc:304] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa60c1192024a6ca11d0c2561fc4619, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:45.917414 30005 raft_consensus.cc:2804] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:45.917480 30005 raft_consensus.cc:493] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:45.917520 30005 raft_consensus.cc:3060] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.918727 30005 raft_consensus.cc:515] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.919178 30005 leader_election.cc:290] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807)
I20260429 19:59:45.919652 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b61d78ffbeb46958a7de3a4abffb632" candidate_uuid: "e1cd42c809074e90a7fc0a3498629296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619"
I20260429 19:59:45.919697 29930 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b61d78ffbeb46958a7de3a4abffb632" candidate_uuid: "e1cd42c809074e90a7fc0a3498629296" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "366cbb43e2504bc19f71a50964fdb6d4"
I20260429 19:59:45.919796 29646 raft_consensus.cc:3060] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.919835 29930 raft_consensus.cc:3060] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:45.920738 29646 raft_consensus.cc:2468] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cd42c809074e90a7fc0a3498629296 in term 1.
I20260429 19:59:45.921025 29930 raft_consensus.cc:2468] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1cd42c809074e90a7fc0a3498629296 in term 1.
I20260429 19:59:45.921108 29720 leader_election.cc:304] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fa60c1192024a6ca11d0c2561fc4619, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:45.921391 30005 raft_consensus.cc:2804] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:45.921805 30005 raft_consensus.cc:697] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 1 LEADER]: Becoming Leader. State: Replica: e1cd42c809074e90a7fc0a3498629296, State: Running, Role: LEADER
I20260429 19:59:45.922103 30005 consensus_queue.cc:237] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:45.925091 29485 catalog_manager.cc:5671] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1cd42c809074e90a7fc0a3498629296 (127.27.17.194). New cstate: current_term: 1 leader_uuid: "e1cd42c809074e90a7fc0a3498629296" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } health_report { overall_health: UNKNOWN } } }
I20260429 19:59:45.975513 29996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.972560 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.27.17.194:39263)
0429 19:59:45.972693 (+ 133us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.972696 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.972709 (+ 13us) server_negotiation.cc:408] Connection header received
0429 19:59:45.972784 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.972786 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.972840 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.972936 (+ 96us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:45.973314 (+ 378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.973794 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.974399 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.974542 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.974595 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.975048 (+ 453us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.975140 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.975294 (+ 154us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.975346 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 19:59:45.985172 30059 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.982318 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.27.17.194:60167)
0429 19:59:45.982621 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:45.982634 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:45.982727 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:45.983020 (+ 293us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:45.983022 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:45.983030 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:45.983274 (+ 244us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.983279 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.983997 (+ 718us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.984000 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:45.984487 (+ 487us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:45.984495 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.984587 (+ 92us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.984628 (+ 41us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.984931 (+ 303us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.984938 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 19:59:45.984990 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 19:59:45.985042 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":112,"threads_started":1}
I20260429 19:59:45.985205 29994 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:45.982454 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:60167 (local address 127.27.17.193:44421)
0429 19:59:45.982601 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:45.982604 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:45.982702 (+ 98us) server_negotiation.cc:408] Connection header received
0429 19:59:45.982817 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:45.982820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:45.982868 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:45.983002 (+ 134us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:45.983418 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.983876 (+ 458us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:45.984603 (+ 727us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:45.984737 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:45.984773 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:45.984841 (+ 68us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:45.984903 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:45.985053 (+ 150us) server_negotiation.cc:300] Negotiation successful
0429 19:59:45.985095 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 19:59:45.991490 29930 raft_consensus.cc:1275] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Refusing update from remote peer e1cd42c809074e90a7fc0a3498629296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:45.991541 29646 raft_consensus.cc:1275] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Refusing update from remote peer e1cd42c809074e90a7fc0a3498629296: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:45.992034 30055 consensus_queue.cc:1048] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:45.992208 30005 consensus_queue.cc:1048] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:46.013042 29996 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:46.009854 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:37423 (local address 127.27.17.194:39263)
0429 19:59:46.009999 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:46.010003 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:46.010014 (+ 11us) server_negotiation.cc:408] Connection header received
0429 19:59:46.010053 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:46.010055 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:46.010130 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:46.010214 (+ 84us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:46.010959 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:46.011516 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:46.012263 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:46.012487 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:46.012548 (+ 61us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:46.012623 (+ 75us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:46.012733 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:46.012889 (+ 156us) server_negotiation.cc:300] Negotiation successful
0429 19:59:46.012933 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 19:59:46.013105 30044 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:46.009453 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:39263 (local address 127.27.17.193:37423)
0429 19:59:46.009617 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:46.009634 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:46.009718 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:46.010262 (+ 544us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:46.010265 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:46.010277 (+ 12us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 19:59:46.010743 (+ 466us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:46.010753 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:46.011660 (+ 907us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:46.011664 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:46.012087 (+ 423us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:46.012094 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:46.012199 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:46.012233 (+ 34us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 19:59:46.012762 (+ 529us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 19:59:46.012767 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 19:59:46.012869 (+ 102us) client_negotiation.cc:241] Negotiation successful
0429 19:59:46.012948 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
W20260429 19:59:46.014043 30028 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260429 19:59:47.577358 29486 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:47.577528 29486 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:47.580044 29486 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Sending DeleteTablet for 3 replicas of tablet 4b61d78ffbeb46958a7de3a4abffb632
I20260429 19:59:47.580682 29626 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b61d78ffbeb46958a7de3a4abffb632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:47 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:45868
I20260429 19:59:47.580772 29768 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b61d78ffbeb46958a7de3a4abffb632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:47 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48076
I20260429 19:59:47.581063 29910 tablet_service.cc:1558] Processing DeleteTablet for tablet 4b61d78ffbeb46958a7de3a4abffb632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:47 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48510
I20260429 19:59:47.581442 30088 tablet_replica.cc:333] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: stopping tablet replica
I20260429 19:59:47.581589 30088 raft_consensus.cc:2243] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:47.581714 30088 raft_consensus.cc:2272] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:47.581938 30089 tablet_replica.cc:333] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: stopping tablet replica
I20260429 19:59:47.582062 30089 raft_consensus.cc:2243] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 1 LEADER]: Raft consensus shutting down.
I20260429 19:59:47.582077 30090 tablet_replica.cc:333] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: stopping tablet replica
I20260429 19:59:47.582418 30089 raft_consensus.cc:2272] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:47.582546 30090 raft_consensus.cc:2243] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:47.582741 30090 raft_consensus.cc:2272] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:47.583549 30089 ts_tablet_manager.cc:1916] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:47.583730 30090 ts_tablet_manager.cc:1916] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:47.585763 30090 ts_tablet_manager.cc:1929] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 19:59:47.585763 30089 ts_tablet_manager.cc:1929] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 19:59:47.585851 30089 log.cc:1199] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/4b61d78ffbeb46958a7de3a4abffb632
I20260429 19:59:47.585851 30090 log.cc:1199] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/4b61d78ffbeb46958a7de3a4abffb632
I20260429 19:59:47.585958 30088 ts_tablet_manager.cc:1916] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:47.586215 30089 ts_tablet_manager.cc:1950] T 4b61d78ffbeb46958a7de3a4abffb632 P e1cd42c809074e90a7fc0a3498629296: Deleting consensus metadata
I20260429 19:59:47.586232 30090 ts_tablet_manager.cc:1950] T 4b61d78ffbeb46958a7de3a4abffb632 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting consensus metadata
I20260429 19:59:47.587035 29471 catalog_manager.cc:5002] TS e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263): tablet 4b61d78ffbeb46958a7de3a4abffb632 (table test-table [id=99369559b5c64ac080f919204edda338]) successfully deleted
I20260429 19:59:47.587035 29472 catalog_manager.cc:5002] TS 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807): tablet 4b61d78ffbeb46958a7de3a4abffb632 (table test-table [id=99369559b5c64ac080f919204edda338]) successfully deleted
I20260429 19:59:47.588243 30088 ts_tablet_manager.cc:1929] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 19:59:47.588310 30088 log.cc:1199] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/4b61d78ffbeb46958a7de3a4abffb632
I20260429 19:59:47.588587 30088 ts_tablet_manager.cc:1950] T 4b61d78ffbeb46958a7de3a4abffb632 P 5fa60c1192024a6ca11d0c2561fc4619: Deleting consensus metadata
I20260429 19:59:47.589243 29472 catalog_manager.cc:5002] TS 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421): tablet 4b61d78ffbeb46958a7de3a4abffb632 (table test-table [id=99369559b5c64ac080f919204edda338]) successfully deleted
Apr 29 19:59:47 dist-test-slave-01nr krb5kdc[29439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492787, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 19:59:47 dist-test-slave-01nr krb5kdc[29439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492787, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:47.614771 30098 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:47.602575 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.27.17.254:35247)
0429 19:59:47.602845 (+ 270us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:47.602849 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:47.602864 (+ 15us) server_negotiation.cc:408] Connection header received
0429 19:59:47.603054 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:47.603057 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:47.603107 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:47.603180 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:47.604162 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:47.604739 (+ 577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:47.605679 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:47.605863 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:47.611463 (+ 5600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:47.611487 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:47.611489 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:47.611523 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:47.613117 (+ 1594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:47.613606 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:47.613612 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:47.613618 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:47.613666 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:47.613913 (+ 247us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:47.613917 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:47.613918 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:47.614079 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:47.614234 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:47.614453 (+ 219us) server_negotiation.cc:300] Negotiation successful
0429 19:59:47.614589 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260429 19:59:47.617393 29486 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 19:59:47.617915 29486 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:47.623122 29626 tablet_service.cc:1511] Processing CreateTablet for tablet cb3d52c003a143349586000b910f874a (DEFAULT_TABLE table=test-table [id=73e5ea7685864a96b9412abdcebff28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:47.623315 29768 tablet_service.cc:1511] Processing CreateTablet for tablet cb3d52c003a143349586000b910f874a (DEFAULT_TABLE table=test-table [id=73e5ea7685864a96b9412abdcebff28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:47.623469 29626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3d52c003a143349586000b910f874a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:47.623494 29910 tablet_service.cc:1511] Processing CreateTablet for tablet cb3d52c003a143349586000b910f874a (DEFAULT_TABLE table=test-table [id=73e5ea7685864a96b9412abdcebff28e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:47.623615 29768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3d52c003a143349586000b910f874a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:47.623773 29910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb3d52c003a143349586000b910f874a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:47.625594 30099 tablet_bootstrap.cc:492] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Bootstrap starting.
I20260429 19:59:47.626006 30100 tablet_bootstrap.cc:492] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Bootstrap starting.
I20260429 19:59:47.626006 30101 tablet_bootstrap.cc:492] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Bootstrap starting.
I20260429 19:59:47.627008 30100 tablet_bootstrap.cc:654] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:47.627008 30101 tablet_bootstrap.cc:654] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:47.627027 30099 tablet_bootstrap.cc:654] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:47.628165 30101 tablet_bootstrap.cc:492] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: No bootstrap required, opened a new log
I20260429 19:59:47.628177 30100 tablet_bootstrap.cc:492] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: No bootstrap required, opened a new log
I20260429 19:59:47.628274 30101 ts_tablet_manager.cc:1403] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260429 19:59:47.628291 30100 ts_tablet_manager.cc:1403] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260429 19:59:47.628801 30100 raft_consensus.cc:359] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.628841 30101 raft_consensus.cc:359] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.628914 30100 raft_consensus.cc:385] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:47.628949 30100 raft_consensus.cc:740] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa60c1192024a6ca11d0c2561fc4619, State: Initialized, Role: FOLLOWER
I20260429 19:59:47.628950 30101 raft_consensus.cc:385] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:47.628985 30101 raft_consensus.cc:740] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Initialized, Role: FOLLOWER
I20260429 19:59:47.629072 30100 consensus_queue.cc:260] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.629101 30101 consensus_queue.cc:260] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.629158 30099 tablet_bootstrap.cc:492] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: No bootstrap required, opened a new log
I20260429 19:59:47.629230 30099 ts_tablet_manager.cc:1403] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260429 19:59:47.629338 30100 ts_tablet_manager.cc:1434] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 19:59:47.629498 30101 ts_tablet_manager.cc:1434] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260429 19:59:47.629673 30099 raft_consensus.cc:359] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.629773 30099 raft_consensus.cc:385] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:47.629810 30099 raft_consensus.cc:740] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cd42c809074e90a7fc0a3498629296, State: Initialized, Role: FOLLOWER
I20260429 19:59:47.629927 30099 consensus_queue.cc:260] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.630192 30099 ts_tablet_manager.cc:1434] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 19:59:47.894837 30105 raft_consensus.cc:493] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:47.895047 30105 raft_consensus.cc:515] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.895658 30105 leader_election.cc:290] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:47.896169 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb3d52c003a143349586000b910f874a" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619" is_pre_election: true
I20260429 19:59:47.896169 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb3d52c003a143349586000b910f874a" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296" is_pre_election: true
I20260429 19:59:47.896353 29646 raft_consensus.cc:2468] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 0.
I20260429 19:59:47.896356 29788 raft_consensus.cc:2468] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 0.
I20260429 19:59:47.896744 29862 leader_election.cc:304] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366cbb43e2504bc19f71a50964fdb6d4, 5fa60c1192024a6ca11d0c2561fc4619; no voters:
I20260429 19:59:47.896937 30105 raft_consensus.cc:2804] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:47.897011 30105 raft_consensus.cc:493] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:47.897066 30105 raft_consensus.cc:3060] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:47.898008 30105 raft_consensus.cc:515] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.898396 30105 leader_election.cc:290] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:47.898842 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb3d52c003a143349586000b910f874a" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296"
I20260429 19:59:47.898962 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb3d52c003a143349586000b910f874a" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619"
I20260429 19:59:47.899021 29788 raft_consensus.cc:3060] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:47.899101 29646 raft_consensus.cc:3060] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:47.899935 29788 raft_consensus.cc:2468] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 1.
I20260429 19:59:47.899963 29646 raft_consensus.cc:2468] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 1.
I20260429 19:59:47.900282 29861 leader_election.cc:304] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366cbb43e2504bc19f71a50964fdb6d4, e1cd42c809074e90a7fc0a3498629296; no voters:
I20260429 19:59:47.900477 30105 raft_consensus.cc:2804] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:47.900733 30105 raft_consensus.cc:697] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Running, Role: LEADER
I20260429 19:59:47.900895 30105 consensus_queue.cc:237] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.902842 29486 catalog_manager.cc:5671] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195). New cstate: current_term: 1 leader_uuid: "366cbb43e2504bc19f71a50964fdb6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } health_report { overall_health: HEALTHY } } }
I20260429 19:59:47.928947 30109 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:47.926032 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:51530 (local address 127.27.17.195:44807)
0429 19:59:47.926346 (+ 314us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:47.926349 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:47.926362 (+ 13us) server_negotiation.cc:408] Connection header received
0429 19:59:47.926401 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:47.926403 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:47.926447 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:47.926526 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 19:59:47.926972 (+ 446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:47.927501 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:47.928249 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:47.928404 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:47.928441 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 19:59:47.928512 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 19:59:47.928576 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:47.928705 (+ 129us) server_negotiation.cc:300] Negotiation successful
0429 19:59:47.928752 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":125,"threads_started":1}
I20260429 19:59:47.935688 29788 raft_consensus.cc:1275] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Refusing update from remote peer 366cbb43e2504bc19f71a50964fdb6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:47.935690 29646 raft_consensus.cc:1275] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Refusing update from remote peer 366cbb43e2504bc19f71a50964fdb6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:47.936178 30106 consensus_queue.cc:1048] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:47.936331 30105 consensus_queue.cc:1048] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:47.969907 29486 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:47.970046 29486 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:47.971963 29486 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Sending DeleteTablet for 3 replicas of tablet cb3d52c003a143349586000b910f874a
I20260429 19:59:47.972474 29626 tablet_service.cc:1558] Processing DeleteTablet for tablet cb3d52c003a143349586000b910f874a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:47 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:45868
I20260429 19:59:47.972589 29768 tablet_service.cc:1558] Processing DeleteTablet for tablet cb3d52c003a143349586000b910f874a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:47 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48076
I20260429 19:59:47.972638 29910 tablet_service.cc:1558] Processing DeleteTablet for tablet cb3d52c003a143349586000b910f874a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:47 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48510
I20260429 19:59:47.972762 30089 tablet_replica.cc:333] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: stopping tablet replica
I20260429 19:59:47.972839 30088 tablet_replica.cc:333] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: stopping tablet replica
I20260429 19:59:47.972918 30089 raft_consensus.cc:2243] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:47.972972 30088 raft_consensus.cc:2243] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:47.973073 30089 raft_consensus.cc:2272] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:47.973109 30088 raft_consensus.cc:2272] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:47.973169 29486 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 19:59:47.973260 30090 tablet_replica.cc:333] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: stopping tablet replica
I20260429 19:59:47.973484 30090 raft_consensus.cc:2243] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 LEADER]: Raft consensus shutting down.
W20260429 19:59:47.973603 29486 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 19:59:47.973707 30090 raft_consensus.cc:2272] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:47.973819 30088 ts_tablet_manager.cc:1916] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:47.974416 30090 ts_tablet_manager.cc:1916] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:47.974591 30089 ts_tablet_manager.cc:1916] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:47.976037 30088 ts_tablet_manager.cc:1929] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:47.976109 30088 log.cc:1199] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/cb3d52c003a143349586000b910f874a
I20260429 19:59:47.976368 30088 ts_tablet_manager.cc:1950] T cb3d52c003a143349586000b910f874a P 5fa60c1192024a6ca11d0c2561fc4619: Deleting consensus metadata
I20260429 19:59:47.976867 30090 ts_tablet_manager.cc:1929] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:47.976946 30090 log.cc:1199] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/cb3d52c003a143349586000b910f874a
I20260429 19:59:47.977157 30089 ts_tablet_manager.cc:1929] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 19:59:47.977233 30089 log.cc:1199] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/cb3d52c003a143349586000b910f874a
I20260429 19:59:47.977432 29472 catalog_manager.cc:5002] TS 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421): tablet cb3d52c003a143349586000b910f874a (table test-table [id=73e5ea7685864a96b9412abdcebff28e]) successfully deleted
I20260429 19:59:47.977533 30089 ts_tablet_manager.cc:1950] T cb3d52c003a143349586000b910f874a P e1cd42c809074e90a7fc0a3498629296: Deleting consensus metadata
I20260429 19:59:47.977655 30090 ts_tablet_manager.cc:1950] T cb3d52c003a143349586000b910f874a P 366cbb43e2504bc19f71a50964fdb6d4: Deleting consensus metadata
I20260429 19:59:47.978349 29471 catalog_manager.cc:5002] TS e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263): tablet cb3d52c003a143349586000b910f874a (table test-table [id=73e5ea7685864a96b9412abdcebff28e]) successfully deleted
I20260429 19:59:47.978555 29472 catalog_manager.cc:5002] TS 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195:44807): tablet cb3d52c003a143349586000b910f874a (table test-table [id=73e5ea7685864a96b9412abdcebff28e]) successfully deleted
I20260429 19:59:47.980706 29626 tablet_service.cc:1511] Processing CreateTablet for tablet 21a39594aa1d419dbe855b882c552157 (DEFAULT_TABLE table=test-table [id=47c32824d84c42d2835790e3afc31557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:47.980777 29768 tablet_service.cc:1511] Processing CreateTablet for tablet 21a39594aa1d419dbe855b882c552157 (DEFAULT_TABLE table=test-table [id=47c32824d84c42d2835790e3afc31557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:47.981006 29768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a39594aa1d419dbe855b882c552157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:47.981007 29626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a39594aa1d419dbe855b882c552157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:47.981009 29910 tablet_service.cc:1511] Processing CreateTablet for tablet 21a39594aa1d419dbe855b882c552157 (DEFAULT_TABLE table=test-table [id=47c32824d84c42d2835790e3afc31557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 19:59:47.981209 29910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21a39594aa1d419dbe855b882c552157. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:47.983175 30101 tablet_bootstrap.cc:492] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Bootstrap starting.
I20260429 19:59:47.983219 30099 tablet_bootstrap.cc:492] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: Bootstrap starting.
I20260429 19:59:47.983994 30099 tablet_bootstrap.cc:654] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:47.983976 30101 tablet_bootstrap.cc:654] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:47.984714 30100 tablet_bootstrap.cc:492] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619: Bootstrap starting.
I20260429 19:59:47.985553 30100 tablet_bootstrap.cc:654] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:47.985576 30101 tablet_bootstrap.cc:492] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: No bootstrap required, opened a new log
I20260429 19:59:47.985644 30101 ts_tablet_manager.cc:1403] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260429 19:59:47.985917 30099 tablet_bootstrap.cc:492] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: No bootstrap required, opened a new log
I20260429 19:59:47.985992 30099 ts_tablet_manager.cc:1403] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260429 19:59:47.985954 30101 raft_consensus.cc:359] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.986073 30101 raft_consensus.cc:385] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:47.986109 30101 raft_consensus.cc:740] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Initialized, Role: FOLLOWER
I20260429 19:59:47.986265 30101 consensus_queue.cc:260] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.986440 30099 raft_consensus.cc:359] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.986533 30099 raft_consensus.cc:385] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:47.986565 30099 raft_consensus.cc:740] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1cd42c809074e90a7fc0a3498629296, State: Initialized, Role: FOLLOWER
I20260429 19:59:47.986569 30101 ts_tablet_manager.cc:1434] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 19:59:47.986717 30100 tablet_bootstrap.cc:492] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619: No bootstrap required, opened a new log
I20260429 19:59:47.986685 30099 consensus_queue.cc:260] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.986788 30100 ts_tablet_manager.cc:1403] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260429 19:59:47.986974 30099 ts_tablet_manager.cc:1434] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 19:59:47.987591 30100 raft_consensus.cc:359] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.987696 30100 raft_consensus.cc:385] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:47.987728 30100 raft_consensus.cc:740] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa60c1192024a6ca11d0c2561fc4619, State: Initialized, Role: FOLLOWER
I20260429 19:59:47.987844 30100 consensus_queue.cc:260] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:47.988096 30100 ts_tablet_manager.cc:1434] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 19:59:48.352470 30113 raft_consensus.cc:493] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 19:59:48.352617 30113 raft_consensus.cc:515] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:48.353246 30113 leader_election.cc:290] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:48.353750 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21a39594aa1d419dbe855b882c552157" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619" is_pre_election: true
I20260429 19:59:48.353756 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21a39594aa1d419dbe855b882c552157" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296" is_pre_election: true
I20260429 19:59:48.353933 29646 raft_consensus.cc:2468] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 0.
I20260429 19:59:48.353937 29788 raft_consensus.cc:2468] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 0.
I20260429 19:59:48.354310 29862 leader_election.cc:304] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366cbb43e2504bc19f71a50964fdb6d4, 5fa60c1192024a6ca11d0c2561fc4619; no voters:
I20260429 19:59:48.354511 30113 raft_consensus.cc:2804] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 19:59:48.354573 30113 raft_consensus.cc:493] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 19:59:48.354625 30113 raft_consensus.cc:3060] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:48.355631 30113 raft_consensus.cc:515] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:48.356078 30113 leader_election.cc:290] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 election: Requested vote from peers 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421), e1cd42c809074e90a7fc0a3498629296 (127.27.17.194:39263)
I20260429 19:59:48.356560 29788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21a39594aa1d419dbe855b882c552157" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1cd42c809074e90a7fc0a3498629296"
I20260429 19:59:48.356588 29646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "21a39594aa1d419dbe855b882c552157" candidate_uuid: "366cbb43e2504bc19f71a50964fdb6d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fa60c1192024a6ca11d0c2561fc4619"
I20260429 19:59:48.356711 29788 raft_consensus.cc:3060] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:48.356727 29646 raft_consensus.cc:3060] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:48.357671 29646 raft_consensus.cc:2468] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 1.
I20260429 19:59:48.357914 29788 raft_consensus.cc:2468] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 366cbb43e2504bc19f71a50964fdb6d4 in term 1.
I20260429 19:59:48.358139 29862 leader_election.cc:304] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 366cbb43e2504bc19f71a50964fdb6d4, 5fa60c1192024a6ca11d0c2561fc4619; no voters:
I20260429 19:59:48.358525 30113 raft_consensus.cc:2804] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:48.358646 30113 raft_consensus.cc:697] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 LEADER]: Becoming Leader. State: Replica: 366cbb43e2504bc19f71a50964fdb6d4, State: Running, Role: LEADER
I20260429 19:59:48.358801 30113 consensus_queue.cc:237] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } }
I20260429 19:59:48.360817 29486 catalog_manager.cc:5671] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 366cbb43e2504bc19f71a50964fdb6d4 (127.27.17.195). New cstate: current_term: 1 leader_uuid: "366cbb43e2504bc19f71a50964fdb6d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "366cbb43e2504bc19f71a50964fdb6d4" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 44807 } health_report { overall_health: HEALTHY } } }
I20260429 19:59:48.449105 29788 raft_consensus.cc:1275] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Refusing update from remote peer 366cbb43e2504bc19f71a50964fdb6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:48.449146 29646 raft_consensus.cc:1275] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Refusing update from remote peer 366cbb43e2504bc19f71a50964fdb6d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 19:59:48.449537 30113 consensus_queue.cc:1048] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1cd42c809074e90a7fc0a3498629296" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 19:59:48.449689 30105 consensus_queue.cc:1048] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fa60c1192024a6ca11d0c2561fc4619" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260429 19:59:48.464133 30070 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20260429 19:59:50.033118 29487 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:50.033283 29487 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 19:59:50.036324 29487 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4020d4127b8246a9bc6ff9352c5e30df: Sending DeleteTablet for 3 replicas of tablet 21a39594aa1d419dbe855b882c552157
I20260429 19:59:50.037104 29626 tablet_service.cc:1558] Processing DeleteTablet for tablet 21a39594aa1d419dbe855b882c552157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:50 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:45868
I20260429 19:59:50.037104 29768 tablet_service.cc:1558] Processing DeleteTablet for tablet 21a39594aa1d419dbe855b882c552157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:50 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48076
I20260429 19:59:50.037163 29910 tablet_service.cc:1558] Processing DeleteTablet for tablet 21a39594aa1d419dbe855b882c552157 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 19:59:50 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48510
I20260429 19:59:50.037479 30143 tablet_replica.cc:333] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619: stopping tablet replica
I20260429 19:59:50.037479 30144 tablet_replica.cc:333] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: stopping tablet replica
I20260429 19:59:50.037619 30144 raft_consensus.cc:2243] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:50.037619 30143 raft_consensus.cc:2243] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 19:59:50.037756 30143 raft_consensus.cc:2272] T 21a39594aa1d419dbe855b882c552157 P 5fa60c1192024a6ca11d0c2561fc4619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:50.037756 30144 raft_consensus.cc:2272] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:50.037755 30145 tablet_replica.cc:333] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: stopping tablet replica
I20260429 19:59:50.037947 30145 raft_consensus.cc:2243] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 LEADER]: Raft consensus shutting down.
I20260429 19:59:50.038036 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 29561
I20260429 19:59:50.039060 30145 raft_consensus.cc:2272] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 19:59:50.039880 30145 ts_tablet_manager.cc:1916] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 19:59:50.040158 30144 ts_tablet_manager.cc:1916] T 21a39594aa1d419dbe855b882c552157 P e1cd42c809074e90a7fc0a3498629296: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260429 19:59:50.047071 29472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.17.193:44421 (error 108)
I20260429 19:59:50.047358 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 29703
I20260429 19:59:50.049193 30147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:50.048714 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:44421 (local address 127.0.0.1:59130)
0429 19:59:50.049032 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:50.049087 (+ 55us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.193:44421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":151,"threads_started":1}
W20260429 19:59:50.049330 29472 catalog_manager.cc:4729] TS 5fa60c1192024a6ca11d0c2561fc4619 (127.27.17.193:44421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21a39594aa1d419dbe855b882c552157: Network error: Client connection negotiation failed: client connection to 127.27.17.193:44421: connect: Connection refused (error 111)
I20260429 19:59:50.049883 30145 ts_tablet_manager.cc:1929] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260429 19:59:50.049999 30145 log.cc:1199] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/21a39594aa1d419dbe855b882c552157
I20260429 19:59:50.050355 30145 ts_tablet_manager.cc:1950] T 21a39594aa1d419dbe855b882c552157 P 366cbb43e2504bc19f71a50964fdb6d4: Deleting consensus metadata
I20260429 19:59:50.055836 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 29845
I20260429 19:59:50.057302 30147 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:50.055803 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:39263 (local address 127.0.0.1:56934)
0429 19:59:50.057153 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:50.057205 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.194:39263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260429 19:59:50.077940 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 29455
2026-04-29T19:59:50Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8156 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:52 dist-test-slave-01nr krb5kdc[30150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:52 dist-test-slave-01nr krb5kdc[30150](info): set up 2 sockets
Apr 29 19:59:52 dist-test-slave-01nr krb5kdc[30150](info): commencing operation
krb5kdc: starting...
W20260429 19:59:54.048085 27719 mini_kdc.cc:121] Time spent starting KDC: real 3.944s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T19:59:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T19:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:54.194367 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34557
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:54.298151 30166 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:54.298411 30166 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:54.298467 30166 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:54.301934 30166 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 19:59:54.302017 30166 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:54.302042 30166 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:54.302060 30166 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 19:59:54.302078 30166 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 19:59:54.307088 30166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:34557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:54.308526 30166 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:54.309468 30166 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:54.316228 30171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:54.316262 30172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:54.316228 30174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:54.316779 30166 server_base.cc:1061] running on GCE node
I20260429 19:59:54.317457 30166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:54.318607 30166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:54.319965 30166 hybrid_clock.cc:648] HybridClock initialized: now 1777492794319915 us; error 59 us; skew 500 ppm
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:54.323061 30166 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 19:59:54.324379 30166 webserver.cc:492] Webserver started at http://127.27.17.254:45095/ using document root <none> and password file <none>
I20260429 19:59:54.325026 30166 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:54.325110 30166 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:54.325354 30166 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:54.327363 30166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "03d183ed669f4fef98db9dccff11c669"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "916752e6197560e9abb92b706fc3cda1"
server_key_iv: "ec62cb5b1eb5c30109b3849d9666a76f"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.327924 30166 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "03d183ed669f4fef98db9dccff11c669"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "916752e6197560e9abb92b706fc3cda1"
server_key_iv: "ec62cb5b1eb5c30109b3849d9666a76f"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.332310 30166 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260429 19:59:54.335018 30181 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.336196 30166 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 19:59:54.336382 30166 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "03d183ed669f4fef98db9dccff11c669"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "916752e6197560e9abb92b706fc3cda1"
server_key_iv: "ec62cb5b1eb5c30109b3849d9666a76f"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.336525 30166 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:54.355535 30166 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:54.362066 30166 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:54.362299 30166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:54.370882 30166 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:34557
I20260429 19:59:54.370946 30233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:34557 every 8 connection(s)
I20260429 19:59:54.372040 30166 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 19:59:54.375010 30234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:54.380641 30234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669: Bootstrap starting.
I20260429 19:59:54.380864 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30166
I20260429 19:59:54.380995 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 19:59:54.381258 27719 external_mini_cluster.cc:1468] Setting key bb4d78cc335f4ac38193015a45e9e78b
I20260429 19:59:54.383898 30234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:54.384866 30234 log.cc:826] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:54.387600 30234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669: No bootstrap required, opened a new log
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:54.391673 30234 raft_consensus.cc:359] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d183ed669f4fef98db9dccff11c669" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34557 } }
I20260429 19:59:54.391914 30234 raft_consensus.cc:385] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:54.392004 30234 raft_consensus.cc:740] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d183ed669f4fef98db9dccff11c669, State: Initialized, Role: FOLLOWER
I20260429 19:59:54.392556 30234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d183ed669f4fef98db9dccff11c669" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34557 } }
I20260429 19:59:54.392715 30234 raft_consensus.cc:399] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 19:59:54.392791 30234 raft_consensus.cc:493] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 19:59:54.392910 30234 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:54.393836 30234 raft_consensus.cc:515] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d183ed669f4fef98db9dccff11c669" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34557 } }
I20260429 19:59:54.394212 30234 leader_election.cc:304] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03d183ed669f4fef98db9dccff11c669; no voters:
I20260429 19:59:54.394562 30234 leader_election.cc:290] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 19:59:54.394640 30239 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:54.394836 30239 raft_consensus.cc:697] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [term 1 LEADER]: Becoming Leader. State: Replica: 03d183ed669f4fef98db9dccff11c669, State: Running, Role: LEADER
I20260429 19:59:54.395211 30239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d183ed669f4fef98db9dccff11c669" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34557 } }
I20260429 19:59:54.396072 30234 sys_catalog.cc:565] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 19:59:54.396636 30241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03d183ed669f4fef98db9dccff11c669. Latest consensus state: current_term: 1 leader_uuid: "03d183ed669f4fef98db9dccff11c669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d183ed669f4fef98db9dccff11c669" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34557 } } }
I20260429 19:59:54.396746 30241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:54.397835 30240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03d183ed669f4fef98db9dccff11c669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03d183ed669f4fef98db9dccff11c669" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34557 } } }
I20260429 19:59:54.398007 30240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:54.398736 30245 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 19:59:54.399006 30237 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:54.382590 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60888 (local address 127.27.17.254:34557)
0429 19:59:54.383077 (+ 487us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:54.383090 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:54.383132 (+ 42us) server_negotiation.cc:408] Connection header received
0429 19:59:54.384158 (+ 1026us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:54.384192 (+ 34us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:54.384633 (+ 441us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:54.385055 (+ 422us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:54.386398 (+ 1343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.387693 (+ 1295us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:54.388910 (+ 1217us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.389270 (+ 360us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:54.391988 (+ 2718us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:54.392022 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:54.392047 (+ 25us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:54.392122 (+ 75us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:54.394965 (+ 2843us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:54.395700 (+ 735us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:54.395705 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:54.395710 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:54.395795 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:54.396122 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:54.396125 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:54.396127 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:54.396607 (+ 480us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:54.396790 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:54.397807 (+ 1017us) server_negotiation.cc:300] Negotiation successful
0429 19:59:54.398083 (+ 276us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":125,"threads_started":1}
I20260429 19:59:54.401973 30245 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 19:59:54.408237 30245 catalog_manager.cc:1357] Generated new cluster ID: 25ce8173ae5b49b7a13f02603a11df82
I20260429 19:59:54.408337 30245 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 19:59:54.423866 30245 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 19:59:54.424808 30245 catalog_manager.cc:1514] Loading token signing keys...
I20260429 19:59:54.434373 30245 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 03d183ed669f4fef98db9dccff11c669: Generated new TSK 0
I20260429 19:59:54.435099 30245 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:54.494560 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34557
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:54.600309 30262 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:54.600533 30262 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:54.600591 30262 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:54.603969 30262 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:54.604048 30262 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:54.604128 30262 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 19:59:54.608604 30262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:54.609695 30262 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:54.610630 30262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:54.617509 30267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:54.617509 30268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:54.617509 30270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:54.617910 30262 server_base.cc:1061] running on GCE node
I20260429 19:59:54.618278 30262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:54.618844 30262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:54.620044 30262 hybrid_clock.cc:648] HybridClock initialized: now 1777492794620004 us; error 50 us; skew 500 ppm
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:54.622761 30262 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 19:59:54.623837 30262 webserver.cc:492] Webserver started at http://127.27.17.193:43255/ using document root <none> and password file <none>
I20260429 19:59:54.624424 30262 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:54.624475 30262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:54.624680 30262 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:54.626384 30262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "e06e1214d1854f9f8e1730b57e5e5d1f"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "092b5c48c584ff2e61ed4e2c51be971c"
server_key_iv: "9db7a1db850dd1f16a8aa2ac34b3047d"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.626866 30262 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "e06e1214d1854f9f8e1730b57e5e5d1f"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "092b5c48c584ff2e61ed4e2c51be971c"
server_key_iv: "9db7a1db850dd1f16a8aa2ac34b3047d"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.630129 30262 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260429 19:59:54.632248 30277 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.633229 30262 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:54.633358 30262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "e06e1214d1854f9f8e1730b57e5e5d1f"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "092b5c48c584ff2e61ed4e2c51be971c"
server_key_iv: "9db7a1db850dd1f16a8aa2ac34b3047d"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.633461 30262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:54.656387 30262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:54.659550 30262 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:54.659788 30262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:54.660426 30262 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:54.661368 30262 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:54.661417 30262 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.661458 30262 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:54.661473 30262 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.671440 30262 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:46167
I20260429 19:59:54.671453 30390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:46167 every 8 connection(s)
I20260429 19:59:54.672444 30262 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 19:59:54.681164 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30262
I20260429 19:59:54.681301 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:54.681612 27719 external_mini_cluster.cc:1468] Setting key 23017662efaed5044bc764067b94bd36
I20260429 19:59:54.686652 30237 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:54.674299 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:36827 (local address 127.27.17.254:34557)
0429 19:59:54.674456 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:54.674459 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:54.675266 (+ 807us) server_negotiation.cc:408] Connection header received
0429 19:59:54.676503 (+ 1237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:54.676507 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:54.676578 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:54.676680 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:54.678240 (+ 1560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.678766 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:54.679510 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.679704 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:54.682410 (+ 2706us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:54.682429 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:54.682432 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:54.682464 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:54.684197 (+ 1733us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:54.684892 (+ 695us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:54.684898 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:54.684901 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:54.684967 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:54.685390 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:54.685396 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:54.685400 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:54.685606 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:54.685712 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:54.686245 (+ 533us) server_negotiation.cc:300] Negotiation successful
0429 19:59:54.686405 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 19:59:54.687508 30393 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:54.674582 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34557 (local address 127.27.17.193:36827)
0429 19:59:54.675095 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:54.675139 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:54.676293 (+ 1154us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:54.676843 (+ 550us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:54.676851 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:54.677232 (+ 381us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:54.678080 (+ 848us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:54.678092 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.678948 (+ 856us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:54.678954 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:54.679384 (+ 430us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:54.679391 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.679623 (+ 232us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:54.680254 (+ 631us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:54.680273 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:54.682191 (+ 1918us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:54.684346 (+ 2155us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:54.684356 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:54.684373 (+ 17us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:54.684766 (+ 393us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:54.685097 (+ 331us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:54.685100 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:54.685102 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:54.685269 (+ 167us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:54.685739 (+ 470us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:54.685745 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:54.685980 (+ 235us) client_negotiation.cc:770] Sending connection context
0429 19:59:54.686169 (+ 189us) client_negotiation.cc:241] Negotiation successful
0429 19:59:54.686448 (+ 279us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":102,"threads_started":1}
I20260429 19:59:54.688875 30391 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34557
I20260429 19:59:54.689297 30391 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:54.689958 30391 heartbeater.cc:507] Master 127.27.17.254:34557 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 19:59:54.691668 30192 ts_manager.cc:194] Registered new tserver with Master: e06e1214d1854f9f8e1730b57e5e5d1f (127.27.17.193:46167)
I20260429 19:59:54.693099 30192 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:36827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:54.738767 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34557
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:54.847831 30398 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:54.848063 30398 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:54.848121 30398 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:54.851568 30398 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:54.851636 30398 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:54.851786 30398 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 19:59:54.856215 30398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:54.857337 30398 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:54.858213 30398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:54.865008 30403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:54.865031 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:54.865031 30404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:54.865200 30398 server_base.cc:1061] running on GCE node
I20260429 19:59:54.865666 30398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:54.866214 30398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:54.867467 30398 hybrid_clock.cc:648] HybridClock initialized: now 1777492794867413 us; error 42 us; skew 500 ppm
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:54.870290 30398 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 19:59:54.871465 30398 webserver.cc:492] Webserver started at http://127.27.17.194:44919/ using document root <none> and password file <none>
I20260429 19:59:54.872044 30398 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:54.872119 30398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:54.872339 30398 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:54.874095 30398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "b17aef010b4f4631bcc84eb8aba6148e"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "f72d110be93273c0dd872a5befe91308"
server_key_iv: "0da1c50a070607cfd4c6d3898f4656b9"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.874605 30398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "b17aef010b4f4631bcc84eb8aba6148e"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "f72d110be93273c0dd872a5befe91308"
server_key_iv: "0da1c50a070607cfd4c6d3898f4656b9"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.878297 30398 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260429 19:59:54.881331 30413 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.882509 30398 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:54.882668 30398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "b17aef010b4f4631bcc84eb8aba6148e"
format_stamp: "Formatted at 2026-04-29 19:59:54 on dist-test-slave-01nr"
server_key: "f72d110be93273c0dd872a5befe91308"
server_key_iv: "0da1c50a070607cfd4c6d3898f4656b9"
server_key_version: "encryptionkey@0"
I20260429 19:59:54.882793 30398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:54.903863 30398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:54.906630 30398 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:54.906843 30398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:54.907466 30398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:54.908381 30398 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:54.908453 30398 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.908524 30398 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:54.908560 30398 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:54.917943 30398 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:35785
I20260429 19:59:54.917976 30526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:35785 every 8 connection(s)
I20260429 19:59:54.919030 30398 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 19:59:54.925498 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30398
I20260429 19:59:54.925593 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 19:59:54.925830 27719 external_mini_cluster.cc:1468] Setting key dd073b21c31859eaf7ad0071c5c33922
Apr 29 19:59:54 dist-test-slave-01nr krb5kdc[30150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492794, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:54.932590 30237 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:54.920913 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:52881 (local address 127.27.17.254:34557)
0429 19:59:54.921103 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:54.921109 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:54.921859 (+ 750us) server_negotiation.cc:408] Connection header received
0429 19:59:54.922684 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:54.922688 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:54.922739 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:54.922818 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:54.924662 (+ 1844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.925306 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:54.926093 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.926292 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:54.928914 (+ 2622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:54.928939 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:54.928942 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:54.928970 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:54.930480 (+ 1510us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:54.931066 (+ 586us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:54.931072 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:54.931078 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:54.931128 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:54.931450 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:54.931453 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:54.931455 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:54.931619 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:54.931723 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:54.932270 (+ 547us) server_negotiation.cc:300] Negotiation successful
0429 19:59:54.932415 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260429 19:59:54.933260 30529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:54.921234 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34557 (local address 127.27.17.194:52881)
0429 19:59:54.921700 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:54.921744 (+ 44us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:54.922494 (+ 750us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:54.923048 (+ 554us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:54.923057 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:54.923512 (+ 455us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:54.924370 (+ 858us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:54.924383 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.925459 (+ 1076us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:54.925462 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:54.925978 (+ 516us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:54.925986 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:54.926153 (+ 167us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:54.926805 (+ 652us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:54.926825 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:54.928742 (+ 1917us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:54.930620 (+ 1878us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:54.930627 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:54.930642 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:54.930953 (+ 311us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:54.931222 (+ 269us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:54.931226 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:54.931228 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:54.931345 (+ 117us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:54.931736 (+ 391us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:54.931744 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:54.931982 (+ 238us) client_negotiation.cc:770] Sending connection context
0429 19:59:54.932221 (+ 239us) client_negotiation.cc:241] Negotiation successful
0429 19:59:54.932470 (+ 249us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":106,"threads_started":1}
I20260429 19:59:54.934473 30527 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34557
I20260429 19:59:54.934756 30527 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 19:59:54.935418 30527 heartbeater.cc:507] Master 127.27.17.254:34557 requested a full tablet report, sending...
I20260429 19:59:54.936518 30192 ts_manager.cc:194] Registered new tserver with Master: b17aef010b4f4631bcc84eb8aba6148e (127.27.17.194:35785)
I20260429 19:59:54.937237 30192 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:52881
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:54.980474 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34557
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:55.083390 30534 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:55.083667 30534 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:55.083758 30534 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:55.087183 30534 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:55.087275 30534 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:55.087400 30534 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 19:59:55.091598 30534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:55.092713 30534 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:55.093581 30534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:55.100339 30542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:55.100339 30539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:55.100358 30540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:55.100564 30534 server_base.cc:1061] running on GCE node
I20260429 19:59:55.101038 30534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:55.101634 30534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:55.102814 30534 hybrid_clock.cc:648] HybridClock initialized: now 1777492795102791 us; error 42 us; skew 500 ppm
Apr 29 19:59:55 dist-test-slave-01nr krb5kdc[30150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492795, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:55.106057 30534 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 19:59:55.107319 30534 webserver.cc:492] Webserver started at http://127.27.17.195:42501/ using document root <none> and password file <none>
I20260429 19:59:55.107908 30534 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:55.107988 30534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:55.108217 30534 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:55.110009 30534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "5e38b391e39342b8b659bdefd074a9b7"
format_stamp: "Formatted at 2026-04-29 19:59:55 on dist-test-slave-01nr"
server_key: "c9c17af4f081fd5363c0917c56443370"
server_key_iv: "8bc01e1b4e7ac0dbe36a9017ee490051"
server_key_version: "encryptionkey@0"
I20260429 19:59:55.110502 30534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "5e38b391e39342b8b659bdefd074a9b7"
format_stamp: "Formatted at 2026-04-29 19:59:55 on dist-test-slave-01nr"
server_key: "c9c17af4f081fd5363c0917c56443370"
server_key_iv: "8bc01e1b4e7ac0dbe36a9017ee490051"
server_key_version: "encryptionkey@0"
I20260429 19:59:55.113938 30534 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 19:59:55.116377 30549 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:55.117420 30534 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 19:59:55.117520 30534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "5e38b391e39342b8b659bdefd074a9b7"
format_stamp: "Formatted at 2026-04-29 19:59:55 on dist-test-slave-01nr"
server_key: "c9c17af4f081fd5363c0917c56443370"
server_key_iv: "8bc01e1b4e7ac0dbe36a9017ee490051"
server_key_version: "encryptionkey@0"
I20260429 19:59:55.117687 30534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:55.139323 30534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:55.142393 30534 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:55.142647 30534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:55.143476 30534 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:55.144418 30534 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:55.144470 30534 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:55.144536 30534 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:55.144567 30534 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:55.155371 30534 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:36663
I20260429 19:59:55.155393 30662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:36663 every 8 connection(s)
I20260429 19:59:55.156445 30534 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 19:59:55.156869 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30534
I20260429 19:59:55.156991 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 19:59:55.157272 27719 external_mini_cluster.cc:1468] Setting key e3eb50dedaabd77949eabb567c6e195a
Apr 29 19:59:55 dist-test-slave-01nr krb5kdc[30150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492795, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:55.169888 30237 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:55.158589 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:55077 (local address 127.27.17.254:34557)
0429 19:59:55.158757 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:55.158761 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:55.159655 (+ 894us) server_negotiation.cc:408] Connection header received
0429 19:59:55.160617 (+ 962us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:55.160621 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:55.160679 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:55.160764 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:55.162397 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:55.162981 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:55.163664 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:55.163849 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:55.166244 (+ 2395us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:55.166274 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:55.166276 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:55.166307 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:55.167758 (+ 1451us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:55.168308 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:55.168311 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:55.168313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:55.168364 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:55.168720 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:55.168723 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:55.168725 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:55.168903 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:55.169018 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:55.169538 (+ 520us) server_negotiation.cc:300] Negotiation successful
0429 19:59:55.169676 (+ 138us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 19:59:55.170603 30665 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:55.159053 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34557 (local address 127.27.17.195:55077)
0429 19:59:55.159511 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:55.159564 (+ 53us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:55.160404 (+ 840us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:55.160947 (+ 543us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:55.160956 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:55.161365 (+ 409us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:55.162238 (+ 873us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:55.162251 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:55.163110 (+ 859us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:55.163114 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:55.163530 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:55.163538 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:55.163780 (+ 242us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:55.164454 (+ 674us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:55.164474 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:55.166059 (+ 1585us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:55.167898 (+ 1839us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:55.167905 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:55.167920 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:55.168179 (+ 259us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:55.168475 (+ 296us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:55.168478 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:55.168480 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:55.168600 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:55.169019 (+ 419us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:55.169026 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:55.169277 (+ 251us) client_negotiation.cc:770] Sending connection context
0429 19:59:55.169498 (+ 221us) client_negotiation.cc:241] Negotiation successful
0429 19:59:55.169766 (+ 268us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":136,"threads_started":1}
I20260429 19:59:55.171770 30663 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34557
I20260429 19:59:55.172029 30663 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:55.172575 30663 heartbeater.cc:507] Master 127.27.17.254:34557 requested a full tablet report, sending...
I20260429 19:59:55.173676 30192 ts_manager.cc:194] Registered new tserver with Master: 5e38b391e39342b8b659bdefd074a9b7 (127.27.17.195:36663)
I20260429 19:59:55.174206 30192 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:55077
I20260429 19:59:55.184072 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 19:59:55.196581 30237 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 19:59:55.195156 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60892 (local address 127.27.17.254:34557)
0429 19:59:55.195389 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:55.195393 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:55.195410 (+ 17us) server_negotiation.cc:408] Connection header received
0429 19:59:55.195568 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:55.195571 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:55.195623 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:55.195705 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:55.196441 (+ 736us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60892: BlockingRecv error: recv got EOF from 127.0.0.1:60892 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260429 19:59:55.198084 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30262
I20260429 19:59:55.204200 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30398
I20260429 19:59:55.210287 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30534
I20260429 19:59:55.216053 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30166
2026-04-29T19:59:55Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5134 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:55 dist-test-slave-01nr krb5kdc[30675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:55 dist-test-slave-01nr krb5kdc[30675](info): set up 2 sockets
Apr 29 19:59:55 dist-test-slave-01nr krb5kdc[30675](info): commencing operation
krb5kdc: starting...
W20260429 19:59:57.252085 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 19:59:57 dist-test-slave-01nr krb5kdc[30675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T19:59:57Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T19:59:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:57.402812 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:38173
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:38173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:57.504666 30691 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:57.504907 30691 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:57.504963 30691 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:57.508450 30691 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 19:59:57.508519 30691 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:57.508543 30691 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:57.508562 30691 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 19:59:57.508580 30691 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 19:59:57.513085 30691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:38173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:38173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:57.514130 30691 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:57.514969 30691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:57.520586 30699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:57.520573 30697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:57.520741 30696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:57.520865 30691 server_base.cc:1061] running on GCE node
I20260429 19:59:57.521337 30691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:57.522257 30691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:57.523411 30691 hybrid_clock.cc:648] HybridClock initialized: now 1777492797523383 us; error 42 us; skew 500 ppm
Apr 29 19:59:57 dist-test-slave-01nr krb5kdc[30675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:57.526144 30691 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 19:59:57.527292 30691 webserver.cc:492] Webserver started at http://127.27.17.254:34435/ using document root <none> and password file <none>
I20260429 19:59:57.527868 30691 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:57.527940 30691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:57.528157 30691 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:57.529803 30691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "2bf3ec40dfa14e85b6251894b5f188c3"
format_stamp: "Formatted at 2026-04-29 19:59:57 on dist-test-slave-01nr"
server_key: "818ddef6eeda8a59d311dec6449cf6b5"
server_key_iv: "900c0e9db9b32b7ab0ff779005ea27ec"
server_key_version: "encryptionkey@0"
I20260429 19:59:57.530273 30691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "2bf3ec40dfa14e85b6251894b5f188c3"
format_stamp: "Formatted at 2026-04-29 19:59:57 on dist-test-slave-01nr"
server_key: "818ddef6eeda8a59d311dec6449cf6b5"
server_key_iv: "900c0e9db9b32b7ab0ff779005ea27ec"
server_key_version: "encryptionkey@0"
I20260429 19:59:57.533578 30691 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 19:59:57.535717 30706 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:57.536703 30691 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260429 19:59:57.536839 30691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "2bf3ec40dfa14e85b6251894b5f188c3"
format_stamp: "Formatted at 2026-04-29 19:59:57 on dist-test-slave-01nr"
server_key: "818ddef6eeda8a59d311dec6449cf6b5"
server_key_iv: "900c0e9db9b32b7ab0ff779005ea27ec"
server_key_version: "encryptionkey@0"
I20260429 19:59:57.536944 30691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:57.557984 30691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:57.560815 30691 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:57.561029 30691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:57.569017 30758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:38173 every 8 connection(s)
I20260429 19:59:57.569025 30691 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:38173
I20260429 19:59:57.570129 30691 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 19:59:57.573035 30759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 19:59:57.578819 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30691
I20260429 19:59:57.578996 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 19:59:57.579023 30759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3: Bootstrap starting.
I20260429 19:59:57.579272 27719 external_mini_cluster.cc:1468] Setting key aba7f4dcc4f0a073f93bf4ec6eb6dc9f
I20260429 19:59:57.581629 30759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3: Neither blocks nor log segments found. Creating new log.
I20260429 19:59:57.582379 30759 log.cc:826] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3: Log is configured to *not* fsync() on all Append() calls
I20260429 19:59:57.584353 30759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3: No bootstrap required, opened a new log
Apr 29 19:59:57 dist-test-slave-01nr krb5kdc[30675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:57.587800 30759 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38173 } }
I20260429 19:59:57.588024 30759 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 19:59:57.588088 30759 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bf3ec40dfa14e85b6251894b5f188c3, State: Initialized, Role: FOLLOWER
I20260429 19:59:57.588528 30759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38173 } }
I20260429 19:59:57.588701 30759 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 19:59:57.588776 30759 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 19:59:57.588907 30759 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 0 FOLLOWER]: Advancing to term 1
I20260429 19:59:57.589823 30759 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38173 } }
I20260429 19:59:57.590216 30759 leader_election.cc:304] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bf3ec40dfa14e85b6251894b5f188c3; no voters:
I20260429 19:59:57.590616 30759 leader_election.cc:290] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 19:59:57.590741 30764 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 19:59:57.591015 30764 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [term 1 LEADER]: Becoming Leader. State: Replica: 2bf3ec40dfa14e85b6251894b5f188c3, State: Running, Role: LEADER
I20260429 19:59:57.591392 30764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38173 } }
I20260429 19:59:57.592046 30759 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 19:59:57.593146 30762 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:57.580612 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44174 (local address 127.27.17.254:38173)
0429 19:59:57.581038 (+ 426us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:57.581047 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:57.581080 (+ 33us) server_negotiation.cc:408] Connection header received
0429 19:59:57.581627 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:57.581645 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:57.581985 (+ 340us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:57.582319 (+ 334us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:57.583424 (+ 1105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:57.584150 (+ 726us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:57.584895 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:57.585178 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:57.587707 (+ 2529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:57.587729 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:57.587741 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:57.587770 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:57.589669 (+ 1899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:57.590152 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:57.590156 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:57.590161 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:57.590233 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:57.590671 (+ 438us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:57.590675 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:57.590677 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:57.591164 (+ 487us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:57.591423 (+ 259us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:57.592054 (+ 631us) server_negotiation.cc:300] Negotiation successful
0429 19:59:57.592327 (+ 273us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":135,"threads_started":1}
I20260429 19:59:57.593168 30766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bf3ec40dfa14e85b6251894b5f188c3. Latest consensus state: current_term: 1 leader_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38173 } } }
I20260429 19:59:57.593302 30766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:57.593559 30765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bf3ec40dfa14e85b6251894b5f188c3" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38173 } } }
I20260429 19:59:57.593638 30765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3 [sys.catalog]: This master's current role is: LEADER
I20260429 19:59:57.593941 30773 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 19:59:57.597041 30773 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 19:59:57.602991 30773 catalog_manager.cc:1357] Generated new cluster ID: 6861b16a54c6430ea4205fffcd61e04d
I20260429 19:59:57.603129 30773 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 19:59:57.615128 30773 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 19:59:57.616014 30773 catalog_manager.cc:1514] Loading token signing keys...
I20260429 19:59:57.629410 30773 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2bf3ec40dfa14e85b6251894b5f188c3: Generated new TSK 0
I20260429 19:59:57.630115 30773 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:57.689950 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:38173
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:57.796133 30787 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:57.796371 30787 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:57.796479 30787 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:57.799932 30787 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:57.800004 30787 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:57.800143 30787 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 19:59:57.804666 30787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:57.806046 30787 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:57.807072 30787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:57.814180 30792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:57.814245 30793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:57.814486 30787 server_base.cc:1061] running on GCE node
W20260429 19:59:57.814523 30795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:57.815074 30787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:57.815692 30787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:57.816871 30787 hybrid_clock.cc:648] HybridClock initialized: now 1777492797816846 us; error 37 us; skew 500 ppm
Apr 29 19:59:57 dist-test-slave-01nr krb5kdc[30675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:57.819806 30787 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 19:59:57.820900 30787 webserver.cc:492] Webserver started at http://127.27.17.193:41617/ using document root <none> and password file <none>
I20260429 19:59:57.821437 30787 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:57.821483 30787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:57.821646 30787 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:57.823386 30787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "1c7e3c4cff084602be781c61d6c59de9"
format_stamp: "Formatted at 2026-04-29 19:59:57 on dist-test-slave-01nr"
server_key: "28b5a4c904b81d0b712cf2f4b2c38887"
server_key_iv: "2d950054278d1868944a122704392c22"
server_key_version: "encryptionkey@0"
I20260429 19:59:57.823830 30787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "1c7e3c4cff084602be781c61d6c59de9"
format_stamp: "Formatted at 2026-04-29 19:59:57 on dist-test-slave-01nr"
server_key: "28b5a4c904b81d0b712cf2f4b2c38887"
server_key_iv: "2d950054278d1868944a122704392c22"
server_key_version: "encryptionkey@0"
I20260429 19:59:57.827395 30787 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 19:59:57.829668 30802 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:57.830770 30787 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:57.830963 30787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "1c7e3c4cff084602be781c61d6c59de9"
format_stamp: "Formatted at 2026-04-29 19:59:57 on dist-test-slave-01nr"
server_key: "28b5a4c904b81d0b712cf2f4b2c38887"
server_key_iv: "2d950054278d1868944a122704392c22"
server_key_version: "encryptionkey@0"
I20260429 19:59:57.831089 30787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:57.843490 30787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:57.846531 30787 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:57.846755 30787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:57.847441 30787 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:57.848368 30787 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:57.848443 30787 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:57.848512 30787 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:57.848555 30787 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:57.858073 30787 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:46591
I20260429 19:59:57.858098 30915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:46591 every 8 connection(s)
I20260429 19:59:57.859074 30787 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 19:59:57.866307 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30787
I20260429 19:59:57.866406 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 19:59:57.866680 27719 external_mini_cluster.cc:1468] Setting key 029f8ee32e9237215b06d8de98e9a2ad
Apr 29 19:59:57 dist-test-slave-01nr krb5kdc[30675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492797, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:57.872390 30762 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:57.860917 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:44415 (local address 127.27.17.254:38173)
0429 19:59:57.861101 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:57.861104 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:57.861792 (+ 688us) server_negotiation.cc:408] Connection header received
0429 19:59:57.862675 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:57.862679 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:57.862736 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:57.862830 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:57.864149 (+ 1319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:57.864732 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:57.865619 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:57.865772 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:57.868768 (+ 2996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:57.868794 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:57.868796 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:57.868824 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:57.870309 (+ 1485us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:57.870856 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:57.870860 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:57.870862 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:57.870978 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:57.871297 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:57.871301 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:57.871303 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:57.871467 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:57.871582 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:57.872085 (+ 503us) server_negotiation.cc:300] Negotiation successful
0429 19:59:57.872215 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260429 19:59:57.873121 30918 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:57.861156 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:38173 (local address 127.27.17.193:44415)
0429 19:59:57.861639 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:57.861673 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:57.862466 (+ 793us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:57.863050 (+ 584us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:57.863058 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:57.863483 (+ 425us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:57.863961 (+ 478us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:57.863973 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:57.865010 (+ 1037us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:57.865014 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:57.865479 (+ 465us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:57.865487 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:57.865658 (+ 171us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:57.866923 (+ 1265us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:57.866944 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:57.868605 (+ 1661us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:57.870449 (+ 1844us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:57.870456 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:57.870470 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:57.870728 (+ 258us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:57.871079 (+ 351us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:57.871082 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:57.871087 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:57.871190 (+ 103us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:57.871583 (+ 393us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:57.871590 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:57.871835 (+ 245us) client_negotiation.cc:770] Sending connection context
0429 19:59:57.872054 (+ 219us) client_negotiation.cc:241] Negotiation successful
0429 19:59:57.872316 (+ 262us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":103,"threads_started":1}
I20260429 19:59:57.874691 30916 heartbeater.cc:344] Connected to a master server at 127.27.17.254:38173
I20260429 19:59:57.875061 30916 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 19:59:57.875860 30916 heartbeater.cc:507] Master 127.27.17.254:38173 requested a full tablet report, sending...
I20260429 19:59:57.877736 30723 ts_manager.cc:194] Registered new tserver with Master: 1c7e3c4cff084602be781c61d6c59de9 (127.27.17.193:46591)
I20260429 19:59:57.878887 30723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:44415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:57.928072 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:38173
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:58.037638 30923 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:58.037891 30923 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:58.037997 30923 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:58.041738 30923 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:58.041816 30923 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:58.041946 30923 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 19:59:58.046586 30923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:58.047829 30923 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:58.048697 30923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:58.055501 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:58.055495 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:58.055495 30931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:58.055975 30923 server_base.cc:1061] running on GCE node
I20260429 19:59:58.056411 30923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:58.056974 30923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:58.058143 30923 hybrid_clock.cc:648] HybridClock initialized: now 1777492798058119 us; error 37 us; skew 500 ppm
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[30675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492798, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:58.061122 30923 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 19:59:58.062227 30923 webserver.cc:492] Webserver started at http://127.27.17.194:45759/ using document root <none> and password file <none>
I20260429 19:59:58.062805 30923 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:58.062879 30923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:58.063104 30923 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:58.064810 30923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "743264c6a29f448d9b83ca777d8e22c1"
format_stamp: "Formatted at 2026-04-29 19:59:58 on dist-test-slave-01nr"
server_key: "0a53c7945829683430571976bbe343dc"
server_key_iv: "4e690d1f55726f250e5c0248c962e59d"
server_key_version: "encryptionkey@0"
I20260429 19:59:58.065303 30923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "743264c6a29f448d9b83ca777d8e22c1"
format_stamp: "Formatted at 2026-04-29 19:59:58 on dist-test-slave-01nr"
server_key: "0a53c7945829683430571976bbe343dc"
server_key_iv: "4e690d1f55726f250e5c0248c962e59d"
server_key_version: "encryptionkey@0"
I20260429 19:59:58.068756 30923 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 19:59:58.071187 30938 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:58.072218 30923 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:58.072386 30923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "743264c6a29f448d9b83ca777d8e22c1"
format_stamp: "Formatted at 2026-04-29 19:59:58 on dist-test-slave-01nr"
server_key: "0a53c7945829683430571976bbe343dc"
server_key_iv: "4e690d1f55726f250e5c0248c962e59d"
server_key_version: "encryptionkey@0"
I20260429 19:59:58.072500 30923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:58.104349 30923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:58.107167 30923 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:58.107372 30923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:58.107949 30923 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:58.108873 30923 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:58.108948 30923 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:58.109018 30923 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:58.109069 30923 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:58.118144 30923 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:43715
I20260429 19:59:58.118168 31051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:43715 every 8 connection(s)
I20260429 19:59:58.119316 30923 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 19:59:58.124899 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 30923
I20260429 19:59:58.125005 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 19:59:58.125254 27719 external_mini_cluster.cc:1468] Setting key 2079edbe7203421e1a7d335c91c969f6
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[30675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492798, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:58.132961 30762 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:58.121086 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:43627 (local address 127.27.17.254:38173)
0429 19:59:58.121276 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:58.121281 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:58.121967 (+ 686us) server_negotiation.cc:408] Connection header received
0429 19:59:58.122787 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:58.122791 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:58.122840 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:58.122989 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:58.124162 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.124773 (+ 611us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:58.125536 (+ 763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.125731 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:58.128691 (+ 2960us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:58.128715 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:58.128717 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:58.128747 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:58.130624 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:58.131373 (+ 749us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:58.131378 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:58.131382 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:58.131452 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:58.131754 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:58.131761 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:58.131765 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:58.131993 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:58.132114 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:58.132614 (+ 500us) server_negotiation.cc:300] Negotiation successful
0429 19:59:58.132762 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260429 19:59:58.133531 31054 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:58.121420 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:38173 (local address 127.27.17.194:43627)
0429 19:59:58.121823 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:58.121856 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:58.122589 (+ 733us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:58.123119 (+ 530us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:58.123126 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:58.123541 (+ 415us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:58.124013 (+ 472us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:58.124024 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.124902 (+ 878us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:58.124904 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:58.125412 (+ 508us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:58.125421 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.125583 (+ 162us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:58.126876 (+ 1293us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:58.126902 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:58.128522 (+ 1620us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:58.130769 (+ 2247us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:58.130775 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:58.130786 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:58.131234 (+ 448us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:58.131559 (+ 325us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:58.131562 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:58.131564 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:58.131654 (+ 90us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:58.132099 (+ 445us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:58.132103 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:58.132351 (+ 248us) client_negotiation.cc:770] Sending connection context
0429 19:59:58.132531 (+ 180us) client_negotiation.cc:241] Negotiation successful
0429 19:59:58.132744 (+ 213us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260429 19:59:58.134649 31052 heartbeater.cc:344] Connected to a master server at 127.27.17.254:38173
I20260429 19:59:58.134970 31052 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 19:59:58.135546 31052 heartbeater.cc:507] Master 127.27.17.254:38173 requested a full tablet report, sending...
I20260429 19:59:58.136667 30723 ts_manager.cc:194] Registered new tserver with Master: 743264c6a29f448d9b83ca777d8e22c1 (127.27.17.194:43715)
I20260429 19:59:58.137235 30723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:43627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 19:59:58.182410 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:38173
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:37863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 19:59:58.290660 31059 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 19:59:58.290977 31059 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 19:59:58.291097 31059 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 19:59:58.294673 31059 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 19:59:58.294764 31059 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 19:59:58.294874 31059 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 19:59:58.299419 31059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:37863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:38173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 19:59:58.300552 31059 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 19:59:58.301453 31059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 19:59:58.308195 31064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 19:59:58.308197 31065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:58.308298 31059 server_base.cc:1061] running on GCE node
W20260429 19:59:58.308197 31067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 19:59:58.308743 31059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 19:59:58.309324 31059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 19:59:58.310510 31059 hybrid_clock.cc:648] HybridClock initialized: now 1777492798310488 us; error 37 us; skew 500 ppm
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[30675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492798, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 19:59:58.313879 31059 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 19:59:58.315140 31059 webserver.cc:492] Webserver started at http://127.27.17.195:40313/ using document root <none> and password file <none>
I20260429 19:59:58.315713 31059 fs_manager.cc:362] Metadata directory not provided
I20260429 19:59:58.315768 31059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 19:59:58.315943 31059 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 19:59:58.317768 31059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "d27bbf1e8d0f42e484bd14c34222a20d"
format_stamp: "Formatted at 2026-04-29 19:59:58 on dist-test-slave-01nr"
server_key: "e571562e30946c57e3aa5496cfb339f2"
server_key_iv: "7ee7cf79eaefe28a02c0369a11314907"
server_key_version: "encryptionkey@0"
I20260429 19:59:58.318260 31059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "d27bbf1e8d0f42e484bd14c34222a20d"
format_stamp: "Formatted at 2026-04-29 19:59:58 on dist-test-slave-01nr"
server_key: "e571562e30946c57e3aa5496cfb339f2"
server_key_iv: "7ee7cf79eaefe28a02c0369a11314907"
server_key_version: "encryptionkey@0"
I20260429 19:59:58.322283 31059 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260429 19:59:58.324841 31074 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:58.325933 31059 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 19:59:58.326076 31059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "d27bbf1e8d0f42e484bd14c34222a20d"
format_stamp: "Formatted at 2026-04-29 19:59:58 on dist-test-slave-01nr"
server_key: "e571562e30946c57e3aa5496cfb339f2"
server_key_iv: "7ee7cf79eaefe28a02c0369a11314907"
server_key_version: "encryptionkey@0"
I20260429 19:59:58.326195 31059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 19:59:58.348593 31059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 19:59:58.351567 31059 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 19:59:58.351807 31059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 19:59:58.352443 31059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 19:59:58.353459 31059 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 19:59:58.353513 31059 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:58.353551 31059 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 19:59:58.353566 31059 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 19:59:58.363943 31059 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:40741
I20260429 19:59:58.363958 31187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:40741 every 8 connection(s)
I20260429 19:59:58.364974 31059 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 19:59:58.368595 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31059
I20260429 19:59:58.368744 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 19:59:58.369061 27719 external_mini_cluster.cc:1468] Setting key cf5b7c041abe467dc9807ebce59913d8
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[30675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492798, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 19:59:58.378376 30762 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:58.366979 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:41815 (local address 127.27.17.254:38173)
0429 19:59:58.367158 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:58.367162 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:58.367889 (+ 727us) server_negotiation.cc:408] Connection header received
0429 19:59:58.368859 (+ 970us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:58.368865 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:58.368934 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:58.369047 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:58.370722 (+ 1675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.371351 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 19:59:58.372039 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.372182 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:58.374988 (+ 2806us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 19:59:58.375004 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 19:59:58.375007 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 19:59:58.375035 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 19:59:58.376569 (+ 1534us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:58.377061 (+ 492us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:58.377063 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:58.377065 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:58.377110 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 19:59:58.377452 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 19:59:58.377455 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 19:59:58.377456 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 19:59:58.377616 (+ 160us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 19:59:58.377694 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
0429 19:59:58.378138 (+ 444us) server_negotiation.cc:300] Negotiation successful
0429 19:59:58.378240 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260429 19:59:58.379133 31190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 19:59:58.367268 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:38173 (local address 127.27.17.195:41815)
0429 19:59:58.367719 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0429 19:59:58.367754 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 19:59:58.368618 (+ 864us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 19:59:58.369323 (+ 705us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 19:59:58.369333 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 19:59:58.369906 (+ 573us) client_negotiation.cc:190] Negotiated authn=SASL
0429 19:59:58.370555 (+ 649us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:58.370570 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.371489 (+ 919us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 19:59:58.371494 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 19:59:58.371921 (+ 427us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 19:59:58.371929 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 19:59:58.372098 (+ 169us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 19:59:58.373141 (+ 1043us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 19:59:58.373161 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 19:59:58.374783 (+ 1622us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 19:59:58.376703 (+ 1920us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:58.376709 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:58.376719 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:58.376964 (+ 245us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:58.377236 (+ 272us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 19:59:58.377239 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 19:59:58.377241 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 19:59:58.377362 (+ 121us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 19:59:58.377705 (+ 343us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 19:59:58.377709 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 19:59:58.377922 (+ 213us) client_negotiation.cc:770] Sending connection context
0429 19:59:58.378094 (+ 172us) client_negotiation.cc:241] Negotiation successful
0429 19:59:58.378287 (+ 193us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":111,"threads_started":1}
I20260429 19:59:58.380208 31188 heartbeater.cc:344] Connected to a master server at 127.27.17.254:38173
I20260429 19:59:58.380450 31188 heartbeater.cc:461] Registering TS with master...
I20260429 19:59:58.380924 31188 heartbeater.cc:507] Master 127.27.17.254:38173 requested a full tablet report, sending...
I20260429 19:59:58.382051 30723 ts_manager.cc:194] Registered new tserver with Master: d27bbf1e8d0f42e484bd14c34222a20d (127.27.17.195:40741)
I20260429 19:59:58.382768 30723 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:41815
I20260429 19:59:58.383410 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 19:59:58.512137 30762 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 19:59:58.508688 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.27.17.254:38173)
0429 19:59:58.508831 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:58.508834 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:58.509746 (+ 912us) server_negotiation.cc:408] Connection header received
0429 19:59:58.510600 (+ 854us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:58.510603 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:58.510653 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:58.510729 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:58.511998 (+ 1269us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44182: BlockingRecv error: recv got EOF from 127.0.0.1:44182 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260429 19:59:58.513592 30762 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 19:59:58.512722 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.27.17.254:38173)
0429 19:59:58.512850 (+ 128us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:58.512856 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:58.512872 (+ 16us) server_negotiation.cc:408] Connection header received
0429 19:59:58.513007 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:58.513010 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:58.513073 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:58.513160 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:58.513493 (+ 333us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44184: BlockingRecv error: recv got EOF from 127.0.0.1:44184 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260429 19:59:58.514653 30762 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 19:59:58.513869 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.27.17.254:38173)
0429 19:59:58.513979 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:58.513983 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:58.514003 (+ 20us) server_negotiation.cc:408] Connection header received
0429 19:59:58.514054 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:58.514056 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:58.514090 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:58.514152 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:58.514549 (+ 397us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44200: BlockingRecv error: recv got EOF from 127.0.0.1:44200 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260429 19:59:58.518433 30762 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 19:59:58.517649 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.27.17.254:38173)
0429 19:59:58.517792 (+ 143us) server_negotiation.cc:207] Beginning negotiation
0429 19:59:58.517799 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 19:59:58.517814 (+ 15us) server_negotiation.cc:408] Connection header received
0429 19:59:58.517844 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 19:59:58.517846 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 19:59:58.517881 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 19:59:58.517964 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
0429 19:59:58.518311 (+ 347us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44208: BlockingRecv error: recv got EOF from 127.0.0.1:44208 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 19:59:58.526851 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30787
I20260429 19:59:58.533210 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30923
I20260429 19:59:58.539783 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31059
I20260429 19:59:58.545501 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 30691
2026-04-29T19:59:58Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (3329 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[31211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[31211](info): set up 2 sockets
Apr 29 19:59:58 dist-test-slave-01nr krb5kdc[31211](info): commencing operation
krb5kdc: starting...
W20260429 20:00:00.580405 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:00 dist-test-slave-01nr krb5kdc[31211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:00.759971 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:38159
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:39533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:38159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:00.879570 31227 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:00.879845 31227 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:00.879943 31227 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:00.884176 31227 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:00.884272 31227 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:00.884305 31227 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:00.884325 31227 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:00.884342 31227 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:00.889871 31227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:38159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:38159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:00.891412 31227 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:00.892449 31227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:00.898922 31235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:00.898881 31232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:00.898976 31227 server_base.cc:1061] running on GCE node
W20260429 20:00:00.898881 31233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:00.899890 31227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:00.900933 31227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:00.902199 31227 hybrid_clock.cc:648] HybridClock initialized: now 1777492800902014 us; error 202 us; skew 500 ppm
Apr 29 20:00:00 dist-test-slave-01nr krb5kdc[31211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492800, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:00.905748 31227 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:00.907158 31227 webserver.cc:492] Webserver started at http://127.27.17.254:40677/ using document root <none> and password file <none>
I20260429 20:00:00.907786 31227 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:00.907866 31227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:00.908090 31227 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:00.909996 31227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "2472f9ef73e74291ac71ce9f32166c6d"
format_stamp: "Formatted at 2026-04-29 20:00:00 on dist-test-slave-01nr"
server_key: "95a19b4d5591f5aaf90bee60423842ae"
server_key_iv: "9536c5a8ab18223aebf3ba712a3c9a49"
server_key_version: "encryptionkey@0"
I20260429 20:00:00.910579 31227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "2472f9ef73e74291ac71ce9f32166c6d"
format_stamp: "Formatted at 2026-04-29 20:00:00 on dist-test-slave-01nr"
server_key: "95a19b4d5591f5aaf90bee60423842ae"
server_key_iv: "9536c5a8ab18223aebf3ba712a3c9a49"
server_key_version: "encryptionkey@0"
I20260429 20:00:00.914412 31227 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260429 20:00:00.917055 31242 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:00.918767 31227 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:00.919466 31227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "2472f9ef73e74291ac71ce9f32166c6d"
format_stamp: "Formatted at 2026-04-29 20:00:00 on dist-test-slave-01nr"
server_key: "95a19b4d5591f5aaf90bee60423842ae"
server_key_iv: "9536c5a8ab18223aebf3ba712a3c9a49"
server_key_version: "encryptionkey@0"
I20260429 20:00:00.919621 31227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:00.958250 31227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:00.961915 31227 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:00.962170 31227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:00.971063 31227 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:38159
I20260429 20:00:00.971057 31294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:38159 every 8 connection(s)
I20260429 20:00:00.972699 31227 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:00.976027 31295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:00.977926 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31227
I20260429 20:00:00.978096 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:00.978399 27719 external_mini_cluster.cc:1468] Setting key bf8bb1677fbbdf80d321c44a68126884
I20260429 20:00:00.983445 31295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d: Bootstrap starting.
I20260429 20:00:00.986390 31295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d: Neither blocks nor log segments found. Creating new log.
Apr 29 20:00:00 dist-test-slave-01nr krb5kdc[31211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:00.987708 31295 log.cc:826] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:00.990738 31295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d: No bootstrap required, opened a new log
I20260429 20:00:00.993413 31298 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:00.980062 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.27.17.254:38159)
0429 20:00:00.980758 (+ 696us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:00.980768 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:00.980804 (+ 36us) server_negotiation.cc:408] Connection header received
0429 20:00:00.981410 (+ 606us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:00.981429 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:00.981762 (+ 333us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:00.982125 (+ 363us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:00.983140 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:00.983979 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:00.984709 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:00.985026 (+ 317us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:00.988131 (+ 3105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:00.988155 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:00.988169 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:00.988205 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:00.990446 (+ 2241us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:00.991106 (+ 660us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:00.991112 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:00.991117 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:00.991213 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:00.991532 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:00.991535 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:00.991537 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:00.991884 (+ 347us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:00.992111 (+ 227us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:00.992438 (+ 327us) server_negotiation.cc:300] Negotiation successful
0429 20:00:00.992714 (+ 276us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":399,"threads_started":1}
I20260429 20:00:00.993808 31295 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2472f9ef73e74291ac71ce9f32166c6d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38159 } }
I20260429 20:00:00.994050 31295 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:00.994133 31295 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2472f9ef73e74291ac71ce9f32166c6d, State: Initialized, Role: FOLLOWER
I20260429 20:00:00.994845 31295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2472f9ef73e74291ac71ce9f32166c6d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38159 } }
I20260429 20:00:00.995054 31295 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:00.995122 31295 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:00.995227 31295 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:00.996404 31295 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2472f9ef73e74291ac71ce9f32166c6d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38159 } }
I20260429 20:00:00.996913 31295 leader_election.cc:304] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2472f9ef73e74291ac71ce9f32166c6d; no voters:
I20260429 20:00:00.997254 27719 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 20:00:00.997344 31295 leader_election.cc:290] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:00.997447 31300 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:00.997658 31300 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [term 1 LEADER]: Becoming Leader. State: Replica: 2472f9ef73e74291ac71ce9f32166c6d, State: Running, Role: LEADER
I20260429 20:00:00.998024 31300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2472f9ef73e74291ac71ce9f32166c6d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38159 } }
I20260429 20:00:00.998672 31295 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:00.999392 31300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2472f9ef73e74291ac71ce9f32166c6d. Latest consensus state: current_term: 1 leader_uuid: "2472f9ef73e74291ac71ce9f32166c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2472f9ef73e74291ac71ce9f32166c6d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38159 } } }
I20260429 20:00:00.999637 31300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:00.999775 31301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2472f9ef73e74291ac71ce9f32166c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2472f9ef73e74291ac71ce9f32166c6d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38159 } } }
I20260429 20:00:00.999897 31301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:01.000218 31305 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:01.003960 31305 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:01.010457 31305 catalog_manager.cc:1357] Generated new cluster ID: 0a1f6896f5714281a7cb48a0293920a7
I20260429 20:00:01.010558 31305 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:01.020874 31298 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:01.014922 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.27.17.254:38159)
0429 20:00:01.015074 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:01.015082 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:01.015539 (+ 457us) server_negotiation.cc:408] Connection header received
0429 20:00:01.015782 (+ 243us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:01.015786 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:01.015845 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:01.015904 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:01.016895 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:01.017627 (+ 732us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:01.018570 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:01.018794 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:01.019377 (+ 583us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:01.019401 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:01.019404 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:01.020293 (+ 889us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:01.020345 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:01.020363 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:01.020490 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:01.020637 (+ 147us) server_negotiation.cc:300] Negotiation successful
0429 20:00:01.020706 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260429 20:00:01.040160 31305 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:01.041790 31305 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:01.057832 31305 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2472f9ef73e74291ac71ce9f32166c6d: Generated new TSK 0
I20260429 20:00:01.058665 31305 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:00:01.073408 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31227
2026-04-29T20:00:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2526 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:01 dist-test-slave-01nr krb5kdc[31328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:01 dist-test-slave-01nr krb5kdc[31328](info): set up 2 sockets
Apr 29 20:00:01 dist-test-slave-01nr krb5kdc[31328](info): commencing operation
krb5kdc: starting...
W20260429 20:00:03.127929 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.033s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:03 dist-test-slave-01nr krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:03.283166 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:41107
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:41107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:03.392889 31344 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:03.393280 31344 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:03.393400 31344 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:03.397140 31344 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:03.397256 31344 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:03.397315 31344 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:03.397360 31344 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:03.397405 31344 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:03.402087 31344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:41107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:41107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:03.403375 31344 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:03.404280 31344 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:03.410624 31349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:03.410633 31352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:03.410823 31350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:03.411054 31344 server_base.cc:1061] running on GCE node
I20260429 20:00:03.411741 31344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:03.412715 31344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:03.413933 31344 hybrid_clock.cc:648] HybridClock initialized: now 1777492803413898 us; error 51 us; skew 500 ppm
Apr 29 20:00:03 dist-test-slave-01nr krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:03.417096 31344 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:03.418697 31344 webserver.cc:492] Webserver started at http://127.27.17.254:39105/ using document root <none> and password file <none>
I20260429 20:00:03.419842 31344 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:03.419948 31344 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:03.420303 31344 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:03.422307 31344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "e36c4ca76fec4743a3c288097842fed8"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "20ca0f656b533f0fd9ad87cd8410627a"
server_key_iv: "d07ed512d5acbae7901857192ce206d0"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.423025 31344 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "e36c4ca76fec4743a3c288097842fed8"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "20ca0f656b533f0fd9ad87cd8410627a"
server_key_iv: "d07ed512d5acbae7901857192ce206d0"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.427408 31344 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260429 20:00:03.429795 31359 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:03.431066 31344 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:03.431242 31344 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "e36c4ca76fec4743a3c288097842fed8"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "20ca0f656b533f0fd9ad87cd8410627a"
server_key_iv: "d07ed512d5acbae7901857192ce206d0"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.431365 31344 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:03.447470 31344 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:03.450851 31344 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:03.451149 31344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:03.459008 31344 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:41107
I20260429 20:00:03.459021 31411 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:41107 every 8 connection(s)
I20260429 20:00:03.460052 31344 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:03.462927 31412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:03.468408 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8: Bootstrap starting.
I20260429 20:00:03.469806 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31344
I20260429 20:00:03.469977 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:03.470227 27719 external_mini_cluster.cc:1468] Setting key 0ae0254f41791525f387ade7ae3a4850
I20260429 20:00:03.471486 31412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:03.472236 31412 log.cc:826] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:03.474572 31412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8: No bootstrap required, opened a new log
I20260429 20:00:03.477429 31412 raft_consensus.cc:359] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36c4ca76fec4743a3c288097842fed8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 41107 } }
I20260429 20:00:03.477650 31412 raft_consensus.cc:385] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:03.477736 31412 raft_consensus.cc:740] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36c4ca76fec4743a3c288097842fed8, State: Initialized, Role: FOLLOWER
Apr 29 20:00:03 dist-test-slave-01nr krb5kdc[31328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:03.478193 31412 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36c4ca76fec4743a3c288097842fed8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 41107 } }
I20260429 20:00:03.478309 31412 raft_consensus.cc:399] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:03.478389 31412 raft_consensus.cc:493] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:03.478503 31412 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:03.479416 31412 raft_consensus.cc:515] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36c4ca76fec4743a3c288097842fed8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 41107 } }
I20260429 20:00:03.479759 31412 leader_election.cc:304] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e36c4ca76fec4743a3c288097842fed8; no voters:
I20260429 20:00:03.480083 31412 leader_election.cc:290] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:03.480165 31417 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:03.480329 31417 raft_consensus.cc:697] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [term 1 LEADER]: Becoming Leader. State: Replica: e36c4ca76fec4743a3c288097842fed8, State: Running, Role: LEADER
I20260429 20:00:03.480599 31417 consensus_queue.cc:237] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36c4ca76fec4743a3c288097842fed8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 41107 } }
I20260429 20:00:03.481271 31412 sys_catalog.cc:565] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:03.481701 31417 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36c4ca76fec4743a3c288097842fed8. Latest consensus state: current_term: 1 leader_uuid: "e36c4ca76fec4743a3c288097842fed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36c4ca76fec4743a3c288097842fed8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 41107 } } }
I20260429 20:00:03.481806 31417 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:03.482357 31418 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e36c4ca76fec4743a3c288097842fed8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36c4ca76fec4743a3c288097842fed8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 41107 } } }
I20260429 20:00:03.482584 31418 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:03.484448 31415 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:03.471458 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41246 (local address 127.27.17.254:41107)
0429 20:00:03.471978 (+ 520us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:03.471988 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:03.472020 (+ 32us) server_negotiation.cc:408] Connection header received
0429 20:00:03.472654 (+ 634us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:03.472673 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:03.472950 (+ 277us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:03.473246 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:03.474167 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:03.475327 (+ 1160us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:03.476026 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:03.476298 (+ 272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:03.478613 (+ 2315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:03.478637 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:03.478655 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:03.478682 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:03.481362 (+ 2680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:03.481889 (+ 527us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:03.481896 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:03.481904 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:03.481992 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:03.482356 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:03.482360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:03.482362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:03.482770 (+ 408us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:03.482980 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:03.483406 (+ 426us) server_negotiation.cc:300] Negotiation successful
0429 20:00:03.483645 (+ 239us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":176,"threads_started":1}
W20260429 20:00:03.486665 31432 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:00:03.486747 31432 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:00:03.486805 31421 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:03.487726 31421 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:03.493098 31421 catalog_manager.cc:1357] Generated new cluster ID: f68dc1aa6dd043b28954f94291b9a5a7
I20260429 20:00:03.493180 31421 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:03.510249 31421 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:03.511529 31421 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:03.528121 31421 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e36c4ca76fec4743a3c288097842fed8: Generated new TSK 0
I20260429 20:00:03.528894 31421 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:03.595525 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:41107
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:03.704481 31440 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:03.704741 31440 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:03.704802 31440 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:03.708446 31440 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:03.708529 31440 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:03.708652 31440 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:03.713675 31440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:03.714884 31440 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:03.715835 31440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:03.722754 31445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:03.722770 31446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:03.722754 31448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:03.723133 31440 server_base.cc:1061] running on GCE node
I20260429 20:00:03.723640 31440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:03.724362 31440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:03.725592 31440 hybrid_clock.cc:648] HybridClock initialized: now 1777492803725571 us; error 44 us; skew 500 ppm
Apr 29 20:00:03 dist-test-slave-01nr krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:03.729032 31440 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:00:03.730388 31440 webserver.cc:492] Webserver started at http://127.27.17.193:39341/ using document root <none> and password file <none>
I20260429 20:00:03.731086 31440 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:03.731163 31440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:03.731348 31440 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:03.733114 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "0828b1c337bb42ca94b76a5ac4745be5"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "e7bc50502a2845e3b916786c803cdde2"
server_key_iv: "8e73b6cdbad6726858aa765ccdf8fd6f"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.733588 31440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "0828b1c337bb42ca94b76a5ac4745be5"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "e7bc50502a2845e3b916786c803cdde2"
server_key_iv: "8e73b6cdbad6726858aa765ccdf8fd6f"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.737354 31440 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260429 20:00:03.739744 31455 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:03.740813 31440 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:03.740911 31440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "0828b1c337bb42ca94b76a5ac4745be5"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "e7bc50502a2845e3b916786c803cdde2"
server_key_iv: "8e73b6cdbad6726858aa765ccdf8fd6f"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.740998 31440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:03.764993 31440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:03.768390 31440 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:03.768625 31440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:03.769610 31440 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:03.770707 31440 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:03.770766 31440 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:03.770845 31440 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:03.770881 31440 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:03.782486 31440 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:42799
I20260429 20:00:03.782609 31568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:42799 every 8 connection(s)
I20260429 20:00:03.783588 31440 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:03.792346 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31440
I20260429 20:00:03.792501 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:03.792793 27719 external_mini_cluster.cc:1468] Setting key cd967a7a00026fc9933c5246aa16f7c8
Apr 29 20:00:03 dist-test-slave-01nr krb5kdc[31328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:03.797940 31415 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:03.785532 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:33825 (local address 127.27.17.254:41107)
0429 20:00:03.785688 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:03.785691 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:03.786509 (+ 818us) server_negotiation.cc:408] Connection header received
0429 20:00:03.787411 (+ 902us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:03.787415 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:03.787469 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:03.787550 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:03.788812 (+ 1262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:03.789358 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:03.790119 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:03.790280 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:03.793757 (+ 3477us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:03.793778 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:03.793780 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:03.793814 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:03.795512 (+ 1698us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:03.796183 (+ 671us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:03.796220 (+ 37us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:03.796221 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:03.796278 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:03.796703 (+ 425us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:03.796709 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:03.796711 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:03.796888 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:03.797004 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:03.797688 (+ 684us) server_negotiation.cc:300] Negotiation successful
0429 20:00:03.797811 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 20:00:03.798947 31571 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:03.785824 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:41107 (local address 127.27.17.193:33825)
0429 20:00:03.786302 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:03.786349 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:03.787186 (+ 837us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:03.787723 (+ 537us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:03.787731 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:03.788131 (+ 400us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:03.788647 (+ 516us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:03.788660 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:03.789535 (+ 875us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:03.789539 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:03.789950 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:03.789957 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:03.790154 (+ 197us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:03.791542 (+ 1388us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:03.791565 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:03.793548 (+ 1983us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:03.795649 (+ 2101us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:03.795660 (+ 11us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:03.795678 (+ 18us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:03.796055 (+ 377us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:03.796424 (+ 369us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:03.796430 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:03.796432 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:03.796573 (+ 141us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:03.797018 (+ 445us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:03.797028 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:03.797345 (+ 317us) client_negotiation.cc:770] Sending connection context
0429 20:00:03.797620 (+ 275us) client_negotiation.cc:241] Negotiation successful
0429 20:00:03.797909 (+ 289us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
I20260429 20:00:03.800237 31569 heartbeater.cc:344] Connected to a master server at 127.27.17.254:41107
I20260429 20:00:03.800554 31569 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:03.801218 31569 heartbeater.cc:507] Master 127.27.17.254:41107 requested a full tablet report, sending...
I20260429 20:00:03.802886 31376 ts_manager.cc:194] Registered new tserver with Master: 0828b1c337bb42ca94b76a5ac4745be5 (127.27.17.193:42799)
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 20:00:03.804177 31376 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:33825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:03.851420 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:41107
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:03.961398 31576 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:03.961746 31576 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:03.961848 31576 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:03.965580 31576 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:03.965723 31576 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:03.965852 31576 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:03.970757 31576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:03.972018 31576 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:03.973240 31576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:03.980320 31584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:03.980338 31581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:03.980320 31582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:03.980690 31576 server_base.cc:1061] running on GCE node
I20260429 20:00:03.981295 31576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:03.982321 31576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:03.983534 31576 hybrid_clock.cc:648] HybridClock initialized: now 1777492803983505 us; error 47 us; skew 500 ppm
Apr 29 20:00:03 dist-test-slave-01nr krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:03.987200 31576 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:00:03.988435 31576 webserver.cc:492] Webserver started at http://127.27.17.194:33219/ using document root <none> and password file <none>
I20260429 20:00:03.989063 31576 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:03.989142 31576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:03.989372 31576 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:03.991261 31576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "9c7edff6541b4135a6290b145365793b"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "817bb8cd7a2bd804cccd22d0fafe40d8"
server_key_iv: "59e410b4df674a4d31f56334a7a75c27"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.991789 31576 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "9c7edff6541b4135a6290b145365793b"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "817bb8cd7a2bd804cccd22d0fafe40d8"
server_key_iv: "59e410b4df674a4d31f56334a7a75c27"
server_key_version: "encryptionkey@0"
I20260429 20:00:03.995745 31576 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260429 20:00:03.998754 31591 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:04.000048 31576 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260429 20:00:04.000212 31576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "9c7edff6541b4135a6290b145365793b"
format_stamp: "Formatted at 2026-04-29 20:00:03 on dist-test-slave-01nr"
server_key: "817bb8cd7a2bd804cccd22d0fafe40d8"
server_key_iv: "59e410b4df674a4d31f56334a7a75c27"
server_key_version: "encryptionkey@0"
I20260429 20:00:04.000349 31576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:04.018328 31576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:04.022291 31576 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:04.022544 31576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:04.023218 31576 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:04.024219 31576 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:04.024276 31576 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:04.024317 31576 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:04.024370 31576 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:04.035844 31576 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:42989
I20260429 20:00:04.035940 31704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:42989 every 8 connection(s)
I20260429 20:00:04.037032 31576 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:04.037941 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31576
I20260429 20:00:04.038064 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:04.038324 27719 external_mini_cluster.cc:1468] Setting key ab5192e75001f22ee6e708fad0d46af2
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492803, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 20:00:04.052775 31415 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:04.039943 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:49027 (local address 127.27.17.254:41107)
0429 20:00:04.041420 (+ 1477us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:04.041426 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:04.041447 (+ 21us) server_negotiation.cc:408] Connection header received
0429 20:00:04.042021 (+ 574us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:04.042026 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:04.042092 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:04.042203 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:04.044207 (+ 2004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.044730 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:04.045560 (+ 830us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.045767 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:04.048437 (+ 2670us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:04.048483 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:04.048486 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:04.048517 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:04.050092 (+ 1575us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:04.050974 (+ 882us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:04.050981 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:04.050987 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:04.051058 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:04.051433 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:04.051437 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:04.051440 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:04.051708 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:04.051835 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:04.052353 (+ 518us) server_negotiation.cc:300] Negotiation successful
0429 20:00:04.052517 (+ 164us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1337}
I20260429 20:00:04.053697 31707 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:04.040397 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:41107 (local address 127.27.17.194:49027)
0429 20:00:04.040922 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:04.040961 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:04.041802 (+ 841us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:04.042446 (+ 644us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:04.042455 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:04.042876 (+ 421us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:04.044016 (+ 1140us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:04.044033 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.044878 (+ 845us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:04.044886 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:04.045332 (+ 446us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:04.045340 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.045700 (+ 360us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:04.046490 (+ 790us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:04.046511 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:04.048227 (+ 1716us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:04.050235 (+ 2008us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:04.050242 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:04.050254 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:04.050773 (+ 519us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:04.051175 (+ 402us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:04.051182 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:04.051185 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:04.051305 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:04.051846 (+ 541us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:04.051852 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:04.052087 (+ 235us) client_negotiation.cc:770] Sending connection context
0429 20:00:04.052270 (+ 183us) client_negotiation.cc:241] Negotiation successful
0429 20:00:04.052497 (+ 227us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":142,"threads_started":1}
I20260429 20:00:04.055101 31705 heartbeater.cc:344] Connected to a master server at 127.27.17.254:41107
I20260429 20:00:04.055398 31705 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:04.056003 31705 heartbeater.cc:507] Master 127.27.17.254:41107 requested a full tablet report, sending...
I20260429 20:00:04.057327 31376 ts_manager.cc:194] Registered new tserver with Master: 9c7edff6541b4135a6290b145365793b (127.27.17.194:42989)
I20260429 20:00:04.058139 31376 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:49027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:04.099617 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:41107
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:04.215149 31712 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:04.215497 31712 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:04.215579 31712 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:04.219241 31712 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:04.219340 31712 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:04.219429 31712 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:04.224411 31712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:41107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:04.225745 31712 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:04.226747 31712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:04.234549 31717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:04.234547 31718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:04.234547 31720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:04.235874 31712 server_base.cc:1061] running on GCE node
I20260429 20:00:04.236383 31712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:04.237048 31712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:04.238219 31712 hybrid_clock.cc:648] HybridClock initialized: now 1777492804238186 us; error 51 us; skew 500 ppm
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492804, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:04.241609 31712 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 20:00:04.242805 31712 webserver.cc:492] Webserver started at http://127.27.17.195:40717/ using document root <none> and password file <none>
I20260429 20:00:04.243453 31712 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:04.243507 31712 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:04.243733 31712 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:04.245765 31712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "ef15a39e6aa34c718df7fdbe7bb98024"
format_stamp: "Formatted at 2026-04-29 20:00:04 on dist-test-slave-01nr"
server_key: "3a2b99ee7c9f1d07e02e5caf282f5e90"
server_key_iv: "4b97d06bfd00224a8261a450c4aedd62"
server_key_version: "encryptionkey@0"
I20260429 20:00:04.246428 31712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "ef15a39e6aa34c718df7fdbe7bb98024"
format_stamp: "Formatted at 2026-04-29 20:00:04 on dist-test-slave-01nr"
server_key: "3a2b99ee7c9f1d07e02e5caf282f5e90"
server_key_iv: "4b97d06bfd00224a8261a450c4aedd62"
server_key_version: "encryptionkey@0"
I20260429 20:00:04.250267 31712 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260429 20:00:04.253192 31727 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:04.254494 31712 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:04.254673 31712 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "ef15a39e6aa34c718df7fdbe7bb98024"
format_stamp: "Formatted at 2026-04-29 20:00:04 on dist-test-slave-01nr"
server_key: "3a2b99ee7c9f1d07e02e5caf282f5e90"
server_key_iv: "4b97d06bfd00224a8261a450c4aedd62"
server_key_version: "encryptionkey@0"
I20260429 20:00:04.254806 31712 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:04.270131 31712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:04.273861 31712 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:04.274109 31712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:04.274984 31712 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:04.275949 31712 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:04.276031 31712 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:04.276104 31712 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:04.276154 31712 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:04.287376 31712 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:40805
I20260429 20:00:04.287585 31840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:40805 every 8 connection(s)
I20260429 20:00:04.288482 31712 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:04.297173 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31712
I20260429 20:00:04.297389 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:04.297734 27719 external_mini_cluster.cc:1468] Setting key 1001b3c456b5372dca047685020574ba
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492804, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:04.303758 31415 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:04.291167 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:35253 (local address 127.27.17.254:41107)
0429 20:00:04.291379 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:04.291382 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:04.292151 (+ 769us) server_negotiation.cc:408] Connection header received
0429 20:00:04.293074 (+ 923us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:04.293079 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:04.293144 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:04.293263 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:04.294859 (+ 1596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.295778 (+ 919us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:04.296496 (+ 718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.296675 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:04.300114 (+ 3439us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:04.300137 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:04.300139 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:04.300169 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:04.301629 (+ 1460us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:04.302222 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:04.302227 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:04.302228 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:04.302280 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:04.302604 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:04.302608 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:04.302610 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:04.302774 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:04.302885 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:04.303442 (+ 557us) server_negotiation.cc:300] Negotiation successful
0429 20:00:04.303588 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260429 20:00:04.304718 31843 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:04.291497 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:41107 (local address 127.27.17.195:35253)
0429 20:00:04.291970 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:04.292007 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:04.292840 (+ 833us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:04.293449 (+ 609us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:04.293458 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:04.293882 (+ 424us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:04.294536 (+ 654us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:04.294554 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.295943 (+ 1389us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:04.295948 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:04.296359 (+ 411us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:04.296367 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.296647 (+ 280us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:04.298036 (+ 1389us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:04.298067 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:04.299934 (+ 1867us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:04.301790 (+ 1856us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:04.301799 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:04.301821 (+ 22us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:04.302090 (+ 269us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:04.302379 (+ 289us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:04.302382 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:04.302387 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:04.302498 (+ 111us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:04.302887 (+ 389us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:04.302925 (+ 38us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:04.303172 (+ 247us) client_negotiation.cc:770] Sending connection context
0429 20:00:04.303399 (+ 227us) client_negotiation.cc:241] Negotiation successful
0429 20:00:04.303661 (+ 262us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260429 20:00:04.306089 31841 heartbeater.cc:344] Connected to a master server at 127.27.17.254:41107
I20260429 20:00:04.306385 31841 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:04.307044 31841 heartbeater.cc:507] Master 127.27.17.254:41107 requested a full tablet report, sending...
I20260429 20:00:04.308262 31376 ts_manager.cc:194] Registered new tserver with Master: ef15a39e6aa34c718df7fdbe7bb98024 (127.27.17.195:40805)
I20260429 20:00:04.308892 31376 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:35253
I20260429 20:00:04.312812 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492804, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:04.339751 31415 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:04.330178 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.27.17.254:41107)
0429 20:00:04.330433 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:04.330462 (+ 29us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:04.330483 (+ 21us) server_negotiation.cc:408] Connection header received
0429 20:00:04.330624 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:04.330627 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:04.330678 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:04.330760 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:04.331881 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.332410 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:04.333172 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:04.333332 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:04.336110 (+ 2778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:04.336136 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:04.336139 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:04.336174 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:04.337752 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:04.338174 (+ 422us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:04.338177 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:04.338178 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:04.338229 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:04.338606 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:04.338610 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:04.338611 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:04.338802 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:04.338994 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:04.339362 (+ 368us) server_negotiation.cc:300] Negotiation successful
0429 20:00:04.339566 (+ 204us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
W20260429 20:00:04.340385 31376 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41252
I20260429 20:00:04.341699 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31440
I20260429 20:00:04.348079 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31576
I20260429 20:00:04.353896 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31712
I20260429 20:00:04.360204 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31344
2026-04-29T20:00:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3289 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31853](info): set up 2 sockets
Apr 29 20:00:04 dist-test-slave-01nr krb5kdc[31853](info): commencing operation
krb5kdc: starting...
W20260429 20:00:06.485365 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.100s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:06 dist-test-slave-01nr krb5kdc[31853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:06.660192 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37541
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:37541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:06.779310 31869 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:06.779633 31869 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:06.779748 31869 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:06.783655 31869 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:06.783806 31869 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:06.783859 31869 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:06.783923 31869 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:06.783972 31869 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:06.788864 31869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:37541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:06.790134 31869 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:06.791126 31869 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:06.797921 31875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:06.797993 31877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:06.798029 31874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:06.798707 31869 server_base.cc:1061] running on GCE node
I20260429 20:00:06.799432 31869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:06.800529 31869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:06.801735 31869 hybrid_clock.cc:648] HybridClock initialized: now 1777492806801708 us; error 46 us; skew 500 ppm
Apr 29 20:00:06 dist-test-slave-01nr krb5kdc[31853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492806, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:06.804662 31869 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:06.805796 31869 webserver.cc:492] Webserver started at http://127.27.17.254:37029/ using document root <none> and password file <none>
I20260429 20:00:06.806358 31869 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:06.806406 31869 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:06.806576 31869 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:06.808822 31869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "a01ad7be8371440ab0a465d47fc7a1b8"
format_stamp: "Formatted at 2026-04-29 20:00:06 on dist-test-slave-01nr"
server_key: "3f787b8d88d57a67a92beed08404d235"
server_key_iv: "1c766f9e3261593bfb940868c140c32d"
server_key_version: "encryptionkey@0"
I20260429 20:00:06.809580 31869 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "a01ad7be8371440ab0a465d47fc7a1b8"
format_stamp: "Formatted at 2026-04-29 20:00:06 on dist-test-slave-01nr"
server_key: "3f787b8d88d57a67a92beed08404d235"
server_key_iv: "1c766f9e3261593bfb940868c140c32d"
server_key_version: "encryptionkey@0"
I20260429 20:00:06.813782 31869 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260429 20:00:06.816690 31884 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:06.818181 31869 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260429 20:00:06.818315 31869 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "a01ad7be8371440ab0a465d47fc7a1b8"
format_stamp: "Formatted at 2026-04-29 20:00:06 on dist-test-slave-01nr"
server_key: "3f787b8d88d57a67a92beed08404d235"
server_key_iv: "1c766f9e3261593bfb940868c140c32d"
server_key_version: "encryptionkey@0"
I20260429 20:00:06.818427 31869 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:06.836629 31869 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:06.839885 31869 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:06.840055 31869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:06.851919 31869 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:37541
I20260429 20:00:06.851959 31936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:37541 every 8 connection(s)
I20260429 20:00:06.853129 31869 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:06.856566 31937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:06.858309 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31869
I20260429 20:00:06.858475 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:06.858745 27719 external_mini_cluster.cc:1468] Setting key 155251a7a2ff504d8301c4faae2ef81f
I20260429 20:00:06.863574 31937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8: Bootstrap starting.
I20260429 20:00:06.866021 31937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:06.867120 31937 log.cc:826] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8: Log is configured to *not* fsync() on all Append() calls
Apr 29 20:00:06 dist-test-slave-01nr krb5kdc[31853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:06.869640 31937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8: No bootstrap required, opened a new log
I20260429 20:00:06.872864 31937 raft_consensus.cc:359] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37541 } }
I20260429 20:00:06.873090 31937 raft_consensus.cc:385] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:06.873128 31937 raft_consensus.cc:740] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a01ad7be8371440ab0a465d47fc7a1b8, State: Initialized, Role: FOLLOWER
I20260429 20:00:06.873590 31937 consensus_queue.cc:260] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37541 } }
I20260429 20:00:06.873713 31937 raft_consensus.cc:399] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:06.873783 31937 raft_consensus.cc:493] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:06.873876 31937 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:06.874015 31940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:06.860258 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.27.17.254:37541)
0429 20:00:06.860763 (+ 505us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:06.860774 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:06.860812 (+ 38us) server_negotiation.cc:408] Connection header received
0429 20:00:06.861629 (+ 817us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:06.861655 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:06.861973 (+ 318us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:06.862346 (+ 373us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:06.863357 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:06.864185 (+ 828us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:06.864864 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:06.865162 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:06.867988 (+ 2826us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:06.868017 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:06.868037 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:06.868078 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:06.870491 (+ 2413us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:06.871174 (+ 683us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:06.871180 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:06.871185 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:06.871282 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:06.871625 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:06.871628 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:06.871629 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:06.871998 (+ 369us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:06.872161 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:06.872587 (+ 426us) server_negotiation.cc:300] Negotiation successful
0429 20:00:06.872901 (+ 314us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":160,"threads_started":1}
I20260429 20:00:06.874945 31937 raft_consensus.cc:515] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37541 } }
I20260429 20:00:06.875365 31937 leader_election.cc:304] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a01ad7be8371440ab0a465d47fc7a1b8; no voters:
I20260429 20:00:06.875700 31937 leader_election.cc:290] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:06.876051 31943 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:06.876310 31943 raft_consensus.cc:697] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [term 1 LEADER]: Becoming Leader. State: Replica: a01ad7be8371440ab0a465d47fc7a1b8, State: Running, Role: LEADER
I20260429 20:00:06.876693 31943 consensus_queue.cc:237] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37541 } }
I20260429 20:00:06.877138 31937 sys_catalog.cc:565] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:06.878314 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a01ad7be8371440ab0a465d47fc7a1b8. Latest consensus state: current_term: 1 leader_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37541 } } }
I20260429 20:00:06.878453 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:06.878880 31947 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:06.878814 31944 sys_catalog.cc:455] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a01ad7be8371440ab0a465d47fc7a1b8" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37541 } } }
I20260429 20:00:06.879017 31944 sys_catalog.cc:458] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:06.882810 31947 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:06.889346 31947 catalog_manager.cc:1357] Generated new cluster ID: c3578a02a51744aa9dfa9b5e52884165
I20260429 20:00:06.889771 31947 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:06.899395 31947 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:06.900346 31947 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:06.910790 31947 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a01ad7be8371440ab0a465d47fc7a1b8: Generated new TSK 0
I20260429 20:00:06.911633 31947 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:06.991737 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37541
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:07.142033 31965 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:07.142552 31965 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:07.142750 31965 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:07.148087 31965 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:07.148200 31965 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:07.148293 31965 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:07.155426 31965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:07.156661 31965 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:07.157552 31965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:07.166146 31971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:07.166149 31973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:07.166237 31970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:07.167002 31965 server_base.cc:1061] running on GCE node
I20260429 20:00:07.167398 31965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:07.167968 31965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:07.169185 31965 hybrid_clock.cc:648] HybridClock initialized: now 1777492807169136 us; error 67 us; skew 500 ppm
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:07.172859 31965 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:00:07.174141 31965 webserver.cc:492] Webserver started at http://127.27.17.193:46643/ using document root <none> and password file <none>
I20260429 20:00:07.174758 31965 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:07.174839 31965 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:07.175103 31965 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:07.177006 31965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "6532dc88bddd41e7a71db57cd6fb4eaa"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "8afdae57e6b4be21bd93dd6835da4248"
server_key_iv: "31e62ff2f78ce65ec1f4c7ce6e136866"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.177572 31965 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "6532dc88bddd41e7a71db57cd6fb4eaa"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "8afdae57e6b4be21bd93dd6835da4248"
server_key_iv: "31e62ff2f78ce65ec1f4c7ce6e136866"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.182337 31965 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260429 20:00:07.187166 31980 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.188570 31965 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260429 20:00:07.188889 31965 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "6532dc88bddd41e7a71db57cd6fb4eaa"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "8afdae57e6b4be21bd93dd6835da4248"
server_key_iv: "31e62ff2f78ce65ec1f4c7ce6e136866"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.189045 31965 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:07.201948 31965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:07.207571 31965 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:07.207778 31965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:07.209635 31965 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:07.211016 31965 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:07.211086 31965 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.211135 31965 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:07.211150 31965 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.224884 31965 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:43407
I20260429 20:00:07.224900 32093 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:43407 every 8 connection(s)
I20260429 20:00:07.226140 31965 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:07.232878 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 31965
I20260429 20:00:07.233031 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:07.233323 27719 external_mini_cluster.cc:1468] Setting key a0d7847dcc9e940b97b9f7421ff06862
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:07.240984 31940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.228258 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:48879 (local address 127.27.17.254:37541)
0429 20:00:07.228485 (+ 227us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:07.228488 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:07.229474 (+ 986us) server_negotiation.cc:408] Connection header received
0429 20:00:07.230415 (+ 941us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:07.230420 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:07.230482 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:07.230599 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:07.232282 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.232808 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.233642 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.233833 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.236610 (+ 2777us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:07.236640 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:07.236643 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:07.236672 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:07.238320 (+ 1648us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.239081 (+ 761us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.239087 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.239089 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.239161 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.239555 (+ 394us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.239558 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.239559 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.239802 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:07.239922 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:07.240714 (+ 792us) server_negotiation.cc:300] Negotiation successful
0429 20:00:07.240844 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260429 20:00:07.242031 32096 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.228559 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37541 (local address 127.27.17.193:48879)
0429 20:00:07.229161 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:07.229195 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:07.230203 (+ 1008us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:07.230770 (+ 567us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:07.230778 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:07.231204 (+ 426us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:07.232096 (+ 892us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:07.232110 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.232949 (+ 839us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.232954 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:07.233505 (+ 551us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:07.233513 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.233767 (+ 254us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.234565 (+ 798us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:07.234592 (+ 27us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:07.236410 (+ 1818us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:07.238569 (+ 2159us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:07.238579 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:07.238592 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:07.238886 (+ 294us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:07.239288 (+ 402us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:07.239292 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:07.239294 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:07.239417 (+ 123us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:07.239953 (+ 536us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:07.239964 (+ 11us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:07.240330 (+ 366us) client_negotiation.cc:770] Sending connection context
0429 20:00:07.240597 (+ 267us) client_negotiation.cc:241] Negotiation successful
0429 20:00:07.240896 (+ 299us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":270,"threads_started":1}
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 20:00:07.243697 32094 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37541
I20260429 20:00:07.244007 32094 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:07.244750 32094 heartbeater.cc:507] Master 127.27.17.254:37541 requested a full tablet report, sending...
I20260429 20:00:07.247470 31901 ts_manager.cc:194] Registered new tserver with Master: 6532dc88bddd41e7a71db57cd6fb4eaa (127.27.17.193:43407)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
I20260429 20:00:07.248747 31901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:48879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:07.300649 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37541
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:07.422376 32101 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:07.422674 32101 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:07.422770 32101 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:07.426476 32101 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:07.426612 32101 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:07.426740 32101 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:07.431476 32101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:07.432646 32101 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:07.433579 32101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:07.440341 32107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:07.440351 32109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:07.440622 32101 server_base.cc:1061] running on GCE node
W20260429 20:00:07.440341 32106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:07.441105 32101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:07.441718 32101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:07.442963 32101 hybrid_clock.cc:648] HybridClock initialized: now 1777492807442928 us; error 38 us; skew 500 ppm
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:07.445721 32101 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:00:07.446889 32101 webserver.cc:492] Webserver started at http://127.27.17.194:35699/ using document root <none> and password file <none>
I20260429 20:00:07.447551 32101 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:07.447628 32101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:07.447842 32101 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:07.449654 32101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "6f534b61fd1f488f992f9d9c087c5263"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "291ae10dfb97b967e8a5c5426625196f"
server_key_iv: "46316791d821417f312a9ea3c03ca410"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.450157 32101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "6f534b61fd1f488f992f9d9c087c5263"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "291ae10dfb97b967e8a5c5426625196f"
server_key_iv: "46316791d821417f312a9ea3c03ca410"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.453639 32101 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 20:00:07.455952 32116 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.456985 32101 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:07.457125 32101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "6f534b61fd1f488f992f9d9c087c5263"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "291ae10dfb97b967e8a5c5426625196f"
server_key_iv: "46316791d821417f312a9ea3c03ca410"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.457276 32101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:07.480204 32101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:07.483034 32101 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:07.483276 32101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:07.483909 32101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:07.484900 32101 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:07.484978 32101 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.485051 32101 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:07.485095 32101 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.495124 32101 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:39897
I20260429 20:00:07.495139 32229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:39897 every 8 connection(s)
I20260429 20:00:07.496125 32101 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:07.498096 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 32101
I20260429 20:00:07.498214 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:07.498523 27719 external_mini_cluster.cc:1468] Setting key 0330cb27d1bd934dc28fef684c0f3345
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 20:00:07.510509 31940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.498101 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:52525 (local address 127.27.17.254:37541)
0429 20:00:07.498333 (+ 232us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:07.498338 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:07.499076 (+ 738us) server_negotiation.cc:408] Connection header received
0429 20:00:07.500013 (+ 937us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:07.500017 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:07.500071 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:07.500163 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:07.502118 (+ 1955us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.502633 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.503612 (+ 979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.503754 (+ 142us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.506611 (+ 2857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:07.506645 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:07.506648 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:07.506679 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:07.508310 (+ 1631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.508898 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.508903 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.508905 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.508966 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.509300 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.509304 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.509307 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.509482 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:07.509596 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:07.510143 (+ 547us) server_negotiation.cc:300] Negotiation successful
0429 20:00:07.510311 (+ 168us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 20:00:07.511226 32232 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.498359 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37541 (local address 127.27.17.194:52525)
0429 20:00:07.498889 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:07.498967 (+ 78us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:07.499755 (+ 788us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:07.500405 (+ 650us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:07.500414 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:07.500807 (+ 393us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:07.501924 (+ 1117us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:07.501945 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.502775 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.502782 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:07.503478 (+ 696us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:07.503489 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.503732 (+ 243us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.504413 (+ 681us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:07.504446 (+ 33us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:07.506403 (+ 1957us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:07.508484 (+ 2081us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:07.508492 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:07.508507 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:07.508776 (+ 269us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:07.509071 (+ 295us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:07.509076 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:07.509079 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:07.509189 (+ 110us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:07.509615 (+ 426us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:07.509622 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:07.509868 (+ 246us) client_negotiation.cc:770] Sending connection context
0429 20:00:07.510093 (+ 225us) client_negotiation.cc:241] Negotiation successful
0429 20:00:07.510351 (+ 258us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":126,"threads_started":1}
I20260429 20:00:07.512387 32230 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37541
I20260429 20:00:07.512637 32230 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:07.513193 32230 heartbeater.cc:507] Master 127.27.17.254:37541 requested a full tablet report, sending...
I20260429 20:00:07.514353 31901 ts_manager.cc:194] Registered new tserver with Master: 6f534b61fd1f488f992f9d9c087c5263 (127.27.17.194:39897)
I20260429 20:00:07.515190 31901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:52525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:07.558722 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37541
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:45829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:07.686787 32237 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:07.687112 32237 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:07.687227 32237 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:07.691381 32237 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:07.691468 32237 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:07.691612 32237 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:07.696763 32237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:45829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:07.697973 32237 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:07.698889 32237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:07.707046 32243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:07.707038 32245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:07.707063 32242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:07.707391 32237 server_base.cc:1061] running on GCE node
I20260429 20:00:07.707857 32237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:07.708490 32237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:07.709697 32237 hybrid_clock.cc:648] HybridClock initialized: now 1777492807709682 us; error 40 us; skew 500 ppm
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:07.712967 32237 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 20:00:07.714236 32237 webserver.cc:492] Webserver started at http://127.27.17.195:36387/ using document root <none> and password file <none>
I20260429 20:00:07.714828 32237 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:07.714939 32237 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:07.715168 32237 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:07.717018 32237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "13f4989241c74a11ad42ed8b970060bd"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "7a025bbede41ae363cade8f6773e411f"
server_key_iv: "78412ad4456639c0f7963bb8fdf54278"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.717546 32237 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "13f4989241c74a11ad42ed8b970060bd"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "7a025bbede41ae363cade8f6773e411f"
server_key_iv: "78412ad4456639c0f7963bb8fdf54278"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.721324 32237 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260429 20:00:07.723847 32252 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.725011 32237 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 20:00:07.725173 32237 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "13f4989241c74a11ad42ed8b970060bd"
format_stamp: "Formatted at 2026-04-29 20:00:07 on dist-test-slave-01nr"
server_key: "7a025bbede41ae363cade8f6773e411f"
server_key_iv: "78412ad4456639c0f7963bb8fdf54278"
server_key_version: "encryptionkey@0"
I20260429 20:00:07.725301 32237 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:07.740541 32237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:07.743645 32237 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:07.743852 32237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:07.744490 32237 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:07.745460 32237 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:07.745525 32237 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.745592 32237 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:07.745626 32237 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:07.757167 32237 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:41511
I20260429 20:00:07.757236 32365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:41511 every 8 connection(s)
I20260429 20:00:07.758195 32237 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:07.765993 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 32237
I20260429 20:00:07.766139 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:07.766441 27719 external_mini_cluster.cc:1468] Setting key 50287194f46b841c1687c2dc5d146b35
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:07.773380 31940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.760430 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:57579 (local address 127.27.17.254:37541)
0429 20:00:07.760611 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:07.760616 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:07.761546 (+ 930us) server_negotiation.cc:408] Connection header received
0429 20:00:07.762817 (+ 1271us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:07.762820 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:07.762874 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:07.763026 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:07.765323 (+ 2297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.765976 (+ 653us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.766768 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.766974 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.769589 (+ 2615us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:07.769614 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:07.769616 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:07.769645 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:07.771359 (+ 1714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.771919 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.771926 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.771928 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.771976 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.772292 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.772301 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.772303 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.772453 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:07.772562 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:07.773059 (+ 497us) server_negotiation.cc:300] Negotiation successful
0429 20:00:07.773199 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 20:00:07.773988 32368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.760731 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37541 (local address 127.27.17.195:57579)
0429 20:00:07.761340 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:07.761389 (+ 49us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:07.762561 (+ 1172us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:07.763233 (+ 672us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:07.763247 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:07.763854 (+ 607us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:07.765117 (+ 1263us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:07.765138 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.766118 (+ 980us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.766122 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:07.766644 (+ 522us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:07.766652 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.766880 (+ 228us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.767607 (+ 727us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:07.767647 (+ 40us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:07.769424 (+ 1777us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:07.771502 (+ 2078us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:07.771509 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:07.771522 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:07.771789 (+ 267us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:07.772067 (+ 278us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:07.772071 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:07.772074 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:07.772180 (+ 106us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:07.772568 (+ 388us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:07.772576 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:07.772813 (+ 237us) client_negotiation.cc:770] Sending connection context
0429 20:00:07.773028 (+ 215us) client_negotiation.cc:241] Negotiation successful
0429 20:00:07.773282 (+ 254us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260429 20:00:07.775149 32366 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37541
I20260429 20:00:07.775488 32366 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:07.776160 32366 heartbeater.cc:507] Master 127.27.17.254:37541 requested a full tablet report, sending...
I20260429 20:00:07.777567 31901 ts_manager.cc:194] Registered new tserver with Master: 13f4989241c74a11ad42ed8b970060bd (127.27.17.195:41511)
I20260429 20:00:07.778244 31901 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:57579
I20260429 20:00:07.781217 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 20:00:07.811298 32373 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.800944 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:59104 (local address 127.27.17.193:43407)
0429 20:00:07.801306 (+ 362us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:07.801314 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:07.801344 (+ 30us) server_negotiation.cc:408] Connection header received
0429 20:00:07.801396 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:07.801400 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:07.801685 (+ 285us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:07.801876 (+ 191us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:07.802749 (+ 873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.803526 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.804218 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.804518 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.807242 (+ 2724us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:07.807272 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:07.807285 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:07.807318 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:07.809241 (+ 1923us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.809894 (+ 653us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.809900 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.809902 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.809960 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.810227 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.810231 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.810233 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.810446 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:07.810675 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:07.810920 (+ 245us) server_negotiation.cc:300] Negotiation successful
0429 20:00:07.811077 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":111,"threads_started":1}
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[31853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:07.822801 31940 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:07.814345 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.27.17.254:37541)
0429 20:00:07.814523 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:07.814528 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:07.814635 (+ 107us) server_negotiation.cc:408] Connection header received
0429 20:00:07.814794 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:07.814798 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:07.814856 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:07.814975 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:07.815723 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.816237 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:07.816888 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:07.817121 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:07.819678 (+ 2557us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:07.819707 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:07.819709 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:07.819737 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:07.821190 (+ 1453us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.821640 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.821644 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.821646 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.821699 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:07.821939 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:07.821943 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:07.821945 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:07.822121 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:07.822266 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:07.822491 (+ 225us) server_negotiation.cc:300] Negotiation successful
0429 20:00:07.822607 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260429 20:00:07.823398 31901 server_base.cc:1143] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45282
I20260429 20:00:07.824194 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31965
I20260429 20:00:07.832917 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 32101
I20260429 20:00:07.839723 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 32237
I20260429 20:00:07.847033 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 31869
2026-04-29T20:00:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3486 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[32378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[32378](info): set up 2 sockets
Apr 29 20:00:07 dist-test-slave-01nr krb5kdc[32378](info): commencing operation
krb5kdc: starting...
W20260429 20:00:09.908298 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.036s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:09 dist-test-slave-01nr krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:10.068104 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34447
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:34447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:10.182569 32394 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:10.182842 32394 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:10.182978 32394 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:10.186622 32394 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:10.186758 32394 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:10.186805 32394 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:10.186867 32394 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:10.186955 32394 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:10.191696 32394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:34447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:10.192930 32394 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:10.193897 32394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:10.199754 32402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:10.199754 32400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:10.199862 32399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:10.200166 32394 server_base.cc:1061] running on GCE node
I20260429 20:00:10.200652 32394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:10.201619 32394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:10.202826 32394 hybrid_clock.cc:648] HybridClock initialized: now 1777492810202800 us; error 46 us; skew 500 ppm
Apr 29 20:00:10 dist-test-slave-01nr krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492810, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:10.206182 32394 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:10.207268 32394 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:10.207369 32394 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:10.207633 32394 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:10.209605 32394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "f7f6b669d1584ad7bda2a8f3679f389e"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "6ce9f5b78863f1948a6fa1c9bc8084c0"
server_key_iv: "b00e5857ccd0eec2fc32cf35c93eb56c"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.210165 32394 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "f7f6b669d1584ad7bda2a8f3679f389e"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "6ce9f5b78863f1948a6fa1c9bc8084c0"
server_key_iv: "b00e5857ccd0eec2fc32cf35c93eb56c"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.214082 32394 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260429 20:00:10.216696 32408 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.217871 32394 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 20:00:10.218037 32394 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "f7f6b669d1584ad7bda2a8f3679f389e"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "6ce9f5b78863f1948a6fa1c9bc8084c0"
server_key_iv: "b00e5857ccd0eec2fc32cf35c93eb56c"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.218165 32394 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:10.230020 32394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:10.232990 32394 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:10.233225 32394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:10.241153 32460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:34447 every 8 connection(s)
I20260429 20:00:10.241153 32394 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:34447
I20260429 20:00:10.241861 32394 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:10.244616 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 32394
I20260429 20:00:10.244714 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:10.244884 32461 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:10.245038 27719 external_mini_cluster.cc:1468] Setting key 46c3df9da249dbbea0458be396aaaeea
I20260429 20:00:10.251740 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e: Bootstrap starting.
Apr 29 20:00:10 dist-test-slave-01nr krb5kdc[32378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:10.254726 32461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:10.255627 32461 log.cc:826] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:10.257814 32461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e: No bootstrap required, opened a new log
I20260429 20:00:10.259995 32464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:10.247256 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.27.17.254:34447)
0429 20:00:10.247655 (+ 399us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:10.247665 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:10.247698 (+ 33us) server_negotiation.cc:408] Connection header received
0429 20:00:10.248265 (+ 567us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:10.248280 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:10.248668 (+ 388us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:10.248962 (+ 294us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:10.250110 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.250968 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:10.251872 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.252230 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:10.254989 (+ 2759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:10.255007 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:10.255019 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:10.255051 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:10.257261 (+ 2210us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:10.257773 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:10.257779 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:10.257783 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:10.257858 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:10.258156 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:10.258160 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:10.258163 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:10.258507 (+ 344us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:10.258722 (+ 215us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:10.259102 (+ 380us) server_negotiation.cc:300] Negotiation successful
0429 20:00:10.259377 (+ 275us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":129,"threads_started":1}
I20260429 20:00:10.260813 32461 raft_consensus.cc:359] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34447 } }
I20260429 20:00:10.261025 32461 raft_consensus.cc:385] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:10.261083 32461 raft_consensus.cc:740] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f6b669d1584ad7bda2a8f3679f389e, State: Initialized, Role: FOLLOWER
I20260429 20:00:10.261499 32461 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34447 } }
I20260429 20:00:10.261626 32461 raft_consensus.cc:399] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:10.261708 32461 raft_consensus.cc:493] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:10.261821 32461 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:10.262825 32461 raft_consensus.cc:515] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34447 } }
I20260429 20:00:10.263221 32461 leader_election.cc:304] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f6b669d1584ad7bda2a8f3679f389e; no voters:
I20260429 20:00:10.263613 32461 leader_election.cc:290] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:10.263996 32466 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:10.264297 32466 raft_consensus.cc:697] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [term 1 LEADER]: Becoming Leader. State: Replica: f7f6b669d1584ad7bda2a8f3679f389e, State: Running, Role: LEADER
I20260429 20:00:10.264598 32466 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34447 } }
I20260429 20:00:10.264849 32461 sys_catalog.cc:565] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:10.266137 32467 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34447 } } }
I20260429 20:00:10.266274 32467 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:10.266564 32468 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7f6b669d1584ad7bda2a8f3679f389e. Latest consensus state: current_term: 1 leader_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f6b669d1584ad7bda2a8f3679f389e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34447 } } }
I20260429 20:00:10.266654 32468 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:10.266995 32475 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:10.269824 32475 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:10.274957 32475 catalog_manager.cc:1357] Generated new cluster ID: 5abc1ce41bcf48c2a48b64c623bb4c10
I20260429 20:00:10.275040 32475 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:10.293658 32475 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:10.294878 32475 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:10.311067 32475 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e: Generated new TSK 0
I20260429 20:00:10.311870 32475 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:10.383903 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34447
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:10.495605 32489 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:10.495850 32489 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:10.495911 32489 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:10.499506 32489 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:10.499580 32489 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:10.499703 32489 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:10.504420 32489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:10.505642 32489 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:10.506570 32489 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:10.514001 32494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:10.514321 32489 server_base.cc:1061] running on GCE node
W20260429 20:00:10.514004 32497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:10.514047 32495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:10.514966 32489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:10.515801 32489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:10.517026 32489 hybrid_clock.cc:648] HybridClock initialized: now 1777492810516990 us; error 45 us; skew 500 ppm
Apr 29 20:00:10 dist-test-slave-01nr krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492810, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:10.520031 32489 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:00:10.520855 32489 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:10.520941 32489 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:10.521171 32489 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:10.523342 32489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "00f3090eef6145a781bf7aac0d54d0c2"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "6abaa2b1cc7390fb8ce641c947654dcd"
server_key_iv: "f793ba55b05d68ee9e529ca5cb38ba63"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.523870 32489 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "00f3090eef6145a781bf7aac0d54d0c2"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "6abaa2b1cc7390fb8ce641c947654dcd"
server_key_iv: "f793ba55b05d68ee9e529ca5cb38ba63"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.527441 32489 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:10.529801 32503 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.531117 32489 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:10.531255 32489 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "00f3090eef6145a781bf7aac0d54d0c2"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "6abaa2b1cc7390fb8ce641c947654dcd"
server_key_iv: "f793ba55b05d68ee9e529ca5cb38ba63"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.531371 32489 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:10.543491 32489 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:10.546777 32489 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:10.547005 32489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:10.547627 32489 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:10.548556 32489 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:10.548628 32489 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.548697 32489 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:10.548745 32489 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.558499 32489 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:35311
I20260429 20:00:10.558519 32616 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:35311 every 8 connection(s)
I20260429 20:00:10.559175 32489 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:10.560397 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 32489
I20260429 20:00:10.560528 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:10.560835 27719 external_mini_cluster.cc:1468] Setting key 4090889be659bad1a6cc6be36d4f67e7
Apr 29 20:00:10 dist-test-slave-01nr krb5kdc[32378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492810, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
I20260429 20:00:10.574373 32464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:10.561141 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:45789 (local address 127.27.17.254:34447)
0429 20:00:10.561297 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:10.561301 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:10.562070 (+ 769us) server_negotiation.cc:408] Connection header received
0429 20:00:10.563508 (+ 1438us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:10.563512 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:10.563572 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:10.563678 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:10.564990 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.565504 (+ 514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:10.566463 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.566621 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:10.569937 (+ 3316us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:10.569962 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:10.569965 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:10.569998 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:10.571793 (+ 1795us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:10.572491 (+ 698us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:10.572495 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:10.572497 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:10.572564 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:10.573076 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:10.573080 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:10.573081 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:10.573259 (+ 178us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:10.573364 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:10.574035 (+ 671us) server_negotiation.cc:300] Negotiation successful
0429 20:00:10.574196 (+ 161us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:10.575212 32619 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:10.561501 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34447 (local address 127.27.17.193:45789)
0429 20:00:10.561929 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:10.561970 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:10.563287 (+ 1317us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:10.563835 (+ 548us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:10.563844 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:10.564211 (+ 367us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:10.564834 (+ 623us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:10.564853 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.565654 (+ 801us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:10.565659 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:10.566344 (+ 685us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:10.566354 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.566613 (+ 259us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:10.567303 (+ 690us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:10.567335 (+ 32us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:10.569691 (+ 2356us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:10.571953 (+ 2262us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:10.571960 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:10.571992 (+ 32us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:10.572315 (+ 323us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:10.572704 (+ 389us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:10.572708 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:10.572709 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:10.572908 (+ 199us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:10.573383 (+ 475us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:10.573389 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:10.573715 (+ 326us) client_negotiation.cc:770] Sending connection context
0429 20:00:10.573952 (+ 237us) client_negotiation.cc:241] Negotiation successful
0429 20:00:10.574205 (+ 253us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":125,"threads_started":1}
I20260429 20:00:10.576586 32617 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34447
I20260429 20:00:10.576853 32617 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:10.577471 32617 heartbeater.cc:507] Master 127.27.17.254:34447 requested a full tablet report, sending...
I20260429 20:00:10.579411 32425 ts_manager.cc:194] Registered new tserver with Master: 00f3090eef6145a781bf7aac0d54d0c2 (127.27.17.193:35311)
I20260429 20:00:10.581156 32425 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:45789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:10.624167 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34447
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:10.734670 32624 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:10.735004 32624 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:10.735107 32624 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:10.738759 32624 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:10.738854 32624 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:10.739042 32624 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:10.743898 32624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:10.745103 32624 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:10.746016 32624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:10.753500 32630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:10.753719 32629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:10.753711 32632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:10.754024 32624 server_base.cc:1061] running on GCE node
I20260429 20:00:10.754513 32624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:10.755276 32624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:10.756538 32624 hybrid_clock.cc:648] HybridClock initialized: now 1777492810756514 us; error 54 us; skew 500 ppm
Apr 29 20:00:10 dist-test-slave-01nr krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492810, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:10.759855 32624 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:00:10.760619 32624 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:10.760680 32624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:10.760850 32624 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:10.762632 32624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "28f61d059a094470be371f45114db575"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "5234b1bb9f32134527d1b4db3d8c256d"
server_key_iv: "607ddb72e8145e9b893cfb1095e776f4"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.763182 32624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "28f61d059a094470be371f45114db575"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "5234b1bb9f32134527d1b4db3d8c256d"
server_key_iv: "607ddb72e8145e9b893cfb1095e776f4"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.766979 32624 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260429 20:00:10.769577 32638 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.770749 32624 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:10.770864 32624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "28f61d059a094470be371f45114db575"
format_stamp: "Formatted at 2026-04-29 20:00:10 on dist-test-slave-01nr"
server_key: "5234b1bb9f32134527d1b4db3d8c256d"
server_key_iv: "607ddb72e8145e9b893cfb1095e776f4"
server_key_version: "encryptionkey@0"
I20260429 20:00:10.771047 32624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:10.788731 32624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:10.791929 32624 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:10.792101 32624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:10.792644 32624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:10.793598 32624 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:10.793651 32624 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.793695 32624 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:10.793710 32624 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:10.804940 32624 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:35601
I20260429 20:00:10.805039 32751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:35601 every 8 connection(s)
I20260429 20:00:10.805629 32624 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:10.811233 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 32624
I20260429 20:00:10.811379 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:10.811652 27719 external_mini_cluster.cc:1468] Setting key 781e9b91b518396f0dfb9ef117a60f47
Apr 29 20:00:10 dist-test-slave-01nr krb5kdc[32378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492810, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 20:00:10.822822 32464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:10.807820 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:34875 (local address 127.27.17.254:34447)
0429 20:00:10.808006 (+ 186us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:10.808010 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:10.809050 (+ 1040us) server_negotiation.cc:408] Connection header received
0429 20:00:10.811170 (+ 2120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:10.811174 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:10.811269 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:10.811409 (+ 140us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:10.812998 (+ 1589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.813975 (+ 977us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:10.814851 (+ 876us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.815126 (+ 275us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:10.818054 (+ 2928us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:10.818088 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:10.818095 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:10.818139 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:10.820229 (+ 2090us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:10.820942 (+ 713us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:10.820946 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:10.820948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:10.821017 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:10.821462 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:10.821467 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:10.821472 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:10.821717 (+ 245us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:10.821846 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:10.822515 (+ 669us) server_negotiation.cc:300] Negotiation successful
0429 20:00:10.822662 (+ 147us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 20:00:10.823654 32754 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:10.808193 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34447 (local address 127.27.17.194:34875)
0429 20:00:10.808852 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:10.808905 (+ 53us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:10.810780 (+ 1875us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:10.811729 (+ 949us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:10.811743 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:10.812296 (+ 553us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:10.812845 (+ 549us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:10.812858 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.814152 (+ 1294us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:10.814156 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:10.814661 (+ 505us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:10.814670 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:10.814879 (+ 209us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:10.815681 (+ 802us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:10.815712 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:10.817838 (+ 2126us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:10.820499 (+ 2661us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:10.820506 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:10.820519 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:10.820797 (+ 278us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:10.821151 (+ 354us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:10.821154 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:10.821156 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:10.821333 (+ 177us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:10.821851 (+ 518us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:10.821856 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:10.822120 (+ 264us) client_negotiation.cc:770] Sending connection context
0429 20:00:10.822450 (+ 330us) client_negotiation.cc:241] Negotiation successful
0429 20:00:10.822696 (+ 246us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":122,"threads_started":1}
I20260429 20:00:10.825155 32752 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34447
I20260429 20:00:10.825587 32752 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 20:00:10.826284 32752 heartbeater.cc:507] Master 127.27.17.254:34447 requested a full tablet report, sending...
I20260429 20:00:10.827811 32425 ts_manager.cc:194] Registered new tserver with Master: 28f61d059a094470be371f45114db575 (127.27.17.194:35601)
I20260429 20:00:10.828522 32425 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:34875
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:10.875924 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34447
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:10.987401 32759 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:10.987639 32759 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:10.987759 32759 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:10.991582 32759 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:10.991658 32759 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:10.991770 32759 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:10.996335 32759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:10.997629 32759 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:10.998564 32759 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:11.005204 32764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.005215 32765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.005383 32759 server_base.cc:1061] running on GCE node
W20260429 20:00:11.005227 32767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.005878 32759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:11.006490 32759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:11.007730 32759 hybrid_clock.cc:648] HybridClock initialized: now 1777492811007700 us; error 45 us; skew 500 ppm
Apr 29 20:00:11 dist-test-slave-01nr krb5kdc[32378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:11.011178 32759 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 20:00:11.012040 32759 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:11.012102 32759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:11.012326 32759 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:11.014140 32759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "d66c70cad951404db2ba2a9e06e37332"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "78902e5eb844b84d1dbb447702095cac"
server_key_iv: "4c25715bb879863c92ad3084b434bce2"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.014655 32759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "d66c70cad951404db2ba2a9e06e37332"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "78902e5eb844b84d1dbb447702095cac"
server_key_iv: "4c25715bb879863c92ad3084b434bce2"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.018193 32759 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:11.020673 305 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.021823 32759 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260429 20:00:11.021968 32759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "d66c70cad951404db2ba2a9e06e37332"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "78902e5eb844b84d1dbb447702095cac"
server_key_iv: "4c25715bb879863c92ad3084b434bce2"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.022084 32759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:11.033175 32759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:11.040160 32759 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:11.040403 32759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:11.041028 32759 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:11.041985 32759 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:11.042034 32759 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.042099 32759 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:11.042131 32759 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.052452 32759 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:34771
I20260429 20:00:11.052673 418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:34771 every 8 connection(s)
I20260429 20:00:11.053120 32759 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
Apr 29 20:00:11 dist-test-slave-01nr krb5kdc[32378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492811, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:11.062678 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 32759
I20260429 20:00:11.062815 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:11.063568 27719 external_mini_cluster.cc:1468] Setting key 52ba0474926e926737916e5d28237686
I20260429 20:00:11.067219 32464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.055201 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:38315 (local address 127.27.17.254:34447)
0429 20:00:11.055367 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.055371 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.056058 (+ 687us) server_negotiation.cc:408] Connection header received
0429 20:00:11.057437 (+ 1379us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.057441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.057522 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.057621 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:11.058789 (+ 1168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.059373 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.060061 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.060238 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.062682 (+ 2444us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:11.062709 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:11.062713 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:11.062749 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:11.064641 (+ 1892us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:11.065358 (+ 717us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:11.065365 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:11.065370 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:11.065449 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:11.065825 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:11.065831 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:11.065833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:11.066088 (+ 255us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:11.066194 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.066871 (+ 677us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.067050 (+ 179us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 20:00:11.068095 421 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.055473 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34447 (local address 127.27.17.195:38315)
0429 20:00:11.055916 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.055948 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.057214 (+ 1266us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.057792 (+ 578us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.057801 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.058161 (+ 360us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:11.058638 (+ 477us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.058649 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.059527 (+ 878us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.059532 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:11.059932 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.059938 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.060159 (+ 221us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.060793 (+ 634us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:11.060812 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:11.062463 (+ 1651us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:11.064795 (+ 2332us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:11.064802 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:11.064817 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:11.065234 (+ 417us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:11.065592 (+ 358us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:11.065598 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:11.065605 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:11.065720 (+ 115us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:11.066194 (+ 474us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:11.066202 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:11.066500 (+ 298us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.066831 (+ 331us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.067128 (+ 297us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260429 20:00:11.069516 419 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34447
I20260429 20:00:11.069800 419 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:11.070351 419 heartbeater.cc:507] Master 127.27.17.254:34447 requested a full tablet report, sending...
I20260429 20:00:11.071744 32425 ts_manager.cc:194] Registered new tserver with Master: d66c70cad951404db2ba2a9e06e37332 (127.27.17.195:34771)
I20260429 20:00:11.072352 32425 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:38315
I20260429 20:00:11.080256 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 20:00:11.090760 32464 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.082792 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.27.17.254:34447)
0429 20:00:11.082986 (+ 194us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.082990 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.083103 (+ 113us) server_negotiation.cc:408] Connection header received
0429 20:00:11.083181 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.083183 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.083248 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.083462 (+ 214us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:11.084435 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.085069 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.085795 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.085956 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.087316 (+ 1360us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:11.087339 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:11.087341 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:11.087372 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:11.088951 (+ 1579us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:11.089445 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:11.089448 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:11.089458 (+ 10us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:11.089515 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:11.089825 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:11.089829 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:11.089830 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:11.090029 (+ 199us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:11.090156 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.090489 (+ 333us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.090599 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 20:00:11.094456 32425 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:11.097121 32425 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:11.112883 430 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.107881 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:35311 (local address 127.0.0.1:60690)
0429 20:00:11.108652 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.108692 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.108862 (+ 170us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.109450 (+ 588us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.109455 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.109475 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:11.109852 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.109864 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.111277 (+ 1413us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.111294 (+ 17us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:11.112103 (+ 809us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.112115 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.112344 (+ 229us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.112414 (+ 70us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.112621 (+ 207us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.112708 (+ 87us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":91,"threads_started":1}
I20260429 20:00:11.113401 431 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.108017 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.27.17.193:35311)
0429 20:00:11.108408 (+ 391us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.108415 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.108866 (+ 451us) server_negotiation.cc:408] Connection header received
0429 20:00:11.109029 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.109036 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.109223 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.109391 (+ 168us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:11.110022 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.111144 (+ 1122us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.112258 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.112812 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.112929 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.113147 (+ 218us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.113247 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":99,"threads_started":1}
I20260429 20:00:11.114578 435 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.109705 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.27.17.195:34771)
0429 20:00:11.110064 (+ 359us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.110070 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.110192 (+ 122us) server_negotiation.cc:408] Connection header received
0429 20:00:11.110311 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.110317 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.110506 (+ 189us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.110643 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:11.111308 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.112438 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.113765 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.114190 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.114281 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.114364 (+ 83us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.114445 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":141,"threads_started":1}
I20260429 20:00:11.113724 432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.108130 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:35601 (local address 127.0.0.1:50100)
0429 20:00:11.109269 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.109280 (+ 11us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.109396 (+ 116us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.110235 (+ 839us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.110240 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.110260 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:11.110531 (+ 271us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.110539 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.112137 (+ 1598us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.112143 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:11.113326 (+ 1183us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.113339 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.113454 (+ 115us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.113469 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.113534 (+ 65us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.113589 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1079,"thread_start_us":88,"threads_started":1}
I20260429 20:00:11.113860 434 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.109545 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:34771 (local address 127.0.0.1:33382)
0429 20:00:11.110093 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.110109 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.110201 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.110802 (+ 601us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.110805 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.110824 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:11.111091 (+ 267us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.111097 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.112587 (+ 1490us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.112591 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:11.113428 (+ 837us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.113439 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.113572 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.113591 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.113643 (+ 52us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.113701 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":114,"threads_started":1}
I20260429 20:00:11.115634 433 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.108691 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.27.17.194:35601)
0429 20:00:11.109084 (+ 393us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.109090 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.109666 (+ 576us) server_negotiation.cc:408] Connection header received
0429 20:00:11.109755 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.109762 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.109924 (+ 162us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.110060 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:11.110670 (+ 610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.111986 (+ 1316us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.114560 (+ 2574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.115115 (+ 555us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.115222 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.115298 (+ 76us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.115379 (+ 81us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260429 20:00:11.117376 32551 tablet_service.cc:1511] Processing CreateTablet for tablet d95eff924d8b4938bedb25c905081476 (DEFAULT_TABLE table=test-table [id=ace5052513a543cab4a32d946b2f2a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:11.118875 32551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95eff924d8b4938bedb25c905081476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:11.118721 32686 tablet_service.cc:1511] Processing CreateTablet for tablet d95eff924d8b4938bedb25c905081476 (DEFAULT_TABLE table=test-table [id=ace5052513a543cab4a32d946b2f2a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:11.118973 353 tablet_service.cc:1511] Processing CreateTablet for tablet d95eff924d8b4938bedb25c905081476 (DEFAULT_TABLE table=test-table [id=ace5052513a543cab4a32d946b2f2a90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:11.119614 32686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95eff924d8b4938bedb25c905081476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:11.119830 353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d95eff924d8b4938bedb25c905081476. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:11.125622 436 tablet_bootstrap.cc:492] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Bootstrap starting.
I20260429 20:00:11.125952 437 tablet_bootstrap.cc:492] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: Bootstrap starting.
I20260429 20:00:11.126884 438 tablet_bootstrap.cc:492] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Bootstrap starting.
I20260429 20:00:11.128100 436 tablet_bootstrap.cc:654] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:11.128175 437 tablet_bootstrap.cc:654] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:11.128908 438 tablet_bootstrap.cc:654] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:11.128841 436 log.cc:826] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:11.128976 437 log.cc:826] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:11.129653 438 log.cc:826] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:11.131022 437 tablet_bootstrap.cc:492] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: No bootstrap required, opened a new log
I20260429 20:00:11.131059 436 tablet_bootstrap.cc:492] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: No bootstrap required, opened a new log
I20260429 20:00:11.131206 437 ts_tablet_manager.cc:1403] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260429 20:00:11.131251 436 ts_tablet_manager.cc:1403] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260429 20:00:11.131444 438 tablet_bootstrap.cc:492] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: No bootstrap required, opened a new log
I20260429 20:00:11.131642 438 ts_tablet_manager.cc:1403] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260429 20:00:11.134063 437 raft_consensus.cc:359] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.134249 437 raft_consensus.cc:385] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:11.134282 437 raft_consensus.cc:740] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f3090eef6145a781bf7aac0d54d0c2, State: Initialized, Role: FOLLOWER
I20260429 20:00:11.134771 437 consensus_queue.cc:260] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.135157 436 raft_consensus.cc:359] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.135393 436 raft_consensus.cc:385] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:11.135454 436 raft_consensus.cc:740] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f61d059a094470be371f45114db575, State: Initialized, Role: FOLLOWER
I20260429 20:00:11.135376 438 raft_consensus.cc:359] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.135648 438 raft_consensus.cc:385] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:11.135701 438 raft_consensus.cc:740] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d66c70cad951404db2ba2a9e06e37332, State: Initialized, Role: FOLLOWER
I20260429 20:00:11.135742 32617 heartbeater.cc:499] Master 127.27.17.254:34447 was elected leader, sending a full tablet report...
I20260429 20:00:11.135844 437 ts_tablet_manager.cc:1434] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260429 20:00:11.135914 436 consensus_queue.cc:260] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.136584 32752 heartbeater.cc:499] Master 127.27.17.254:34447 was elected leader, sending a full tablet report...
I20260429 20:00:11.136862 438 consensus_queue.cc:260] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.137550 436 ts_tablet_manager.cc:1434] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260429 20:00:11.137774 438 ts_tablet_manager.cc:1434] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260429 20:00:11.138082 419 heartbeater.cc:499] Master 127.27.17.254:34447 was elected leader, sending a full tablet report...
I20260429 20:00:11.146023 442 raft_consensus.cc:493] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:11.146217 442 raft_consensus.cc:515] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.147456 442 leader_election.cc:290] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28f61d059a094470be371f45114db575 (127.27.17.194:35601), d66c70cad951404db2ba2a9e06e37332 (127.27.17.195:34771)
I20260429 20:00:11.150816 446 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.147745 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:34771 (local address 127.27.17.193:42127)
0429 20:00:11.148068 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.148080 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.148207 (+ 127us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.148637 (+ 430us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.148640 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.148693 (+ 53us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:11.148929 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.148934 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.149742 (+ 808us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.149746 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:11.150473 (+ 727us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.150481 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.150594 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.150609 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.150648 (+ 39us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.150691 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"spinlock_wait_cycles":1280,"thread_start_us":119,"threads_started":1}
I20260429 20:00:11.151429 435 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.147849 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:42127 (local address 127.27.17.195:34771)
0429 20:00:11.147981 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.147986 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.148197 (+ 211us) server_negotiation.cc:408] Connection header received
0429 20:00:11.148340 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.148343 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.148389 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.148611 (+ 222us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:11.149077 (+ 466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.149569 (+ 492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.150631 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.151166 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.151203 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.151252 (+ 49us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.151311 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 20:00:11.151906 445 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.147627 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:35601 (local address 127.27.17.193:35865)
0429 20:00:11.148041 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.148054 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.148184 (+ 130us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.148628 (+ 444us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.148632 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.148651 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:11.148917 (+ 266us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.148923 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.150079 (+ 1156us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.150082 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:11.150838 (+ 756us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:11.150847 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.151628 (+ 781us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.151640 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.151735 (+ 95us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.151791 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":137,"threads_started":1}
I20260429 20:00:11.151928 433 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.147764 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:35865 (local address 127.27.17.194:35601)
0429 20:00:11.147920 (+ 156us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.147924 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.148143 (+ 219us) server_negotiation.cc:408] Connection header received
0429 20:00:11.148391 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.148399 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.148480 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.148569 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:11.149042 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.149921 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.151029 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.151522 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.151555 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.151766 (+ 211us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.151830 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 20:00:11.152097 373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d95eff924d8b4938bedb25c905081476" candidate_uuid: "00f3090eef6145a781bf7aac0d54d0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66c70cad951404db2ba2a9e06e37332" is_pre_election: true
I20260429 20:00:11.152484 373 raft_consensus.cc:2468] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f3090eef6145a781bf7aac0d54d0c2 in term 0.
I20260429 20:00:11.152482 32706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d95eff924d8b4938bedb25c905081476" candidate_uuid: "00f3090eef6145a781bf7aac0d54d0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f61d059a094470be371f45114db575" is_pre_election: true
I20260429 20:00:11.152758 32706 raft_consensus.cc:2468] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00f3090eef6145a781bf7aac0d54d0c2 in term 0.
I20260429 20:00:11.153043 32504 leader_election.cc:304] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f3090eef6145a781bf7aac0d54d0c2, d66c70cad951404db2ba2a9e06e37332; no voters:
I20260429 20:00:11.153329 442 raft_consensus.cc:2804] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:11.153406 442 raft_consensus.cc:493] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:11.153447 442 raft_consensus.cc:3060] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:11.154608 442 raft_consensus.cc:515] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.155098 442 leader_election.cc:290] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 28f61d059a094470be371f45114db575 (127.27.17.194:35601), d66c70cad951404db2ba2a9e06e37332 (127.27.17.195:34771)
I20260429 20:00:11.155529 32706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d95eff924d8b4938bedb25c905081476" candidate_uuid: "00f3090eef6145a781bf7aac0d54d0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28f61d059a094470be371f45114db575"
I20260429 20:00:11.155603 373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d95eff924d8b4938bedb25c905081476" candidate_uuid: "00f3090eef6145a781bf7aac0d54d0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d66c70cad951404db2ba2a9e06e37332"
I20260429 20:00:11.155702 32706 raft_consensus.cc:3060] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:11.155750 373 raft_consensus.cc:3060] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:11.156917 32706 raft_consensus.cc:2468] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f3090eef6145a781bf7aac0d54d0c2 in term 1.
I20260429 20:00:11.156924 373 raft_consensus.cc:2468] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00f3090eef6145a781bf7aac0d54d0c2 in term 1.
I20260429 20:00:11.157263 32507 leader_election.cc:304] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00f3090eef6145a781bf7aac0d54d0c2, 28f61d059a094470be371f45114db575; no voters:
I20260429 20:00:11.157442 442 raft_consensus.cc:2804] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:11.157708 442 raft_consensus.cc:697] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 00f3090eef6145a781bf7aac0d54d0c2, State: Running, Role: LEADER
I20260429 20:00:11.158077 442 consensus_queue.cc:237] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } }
I20260429 20:00:11.161479 32424 catalog_manager.cc:5671] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00f3090eef6145a781bf7aac0d54d0c2 (127.27.17.193). New cstate: current_term: 1 leader_uuid: "00f3090eef6145a781bf7aac0d54d0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00f3090eef6145a781bf7aac0d54d0c2" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 35311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:11.177695 431 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.174558 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.27.17.193:35311)
0429 20:00:11.174705 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.174709 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.174721 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:11.174804 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.174807 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.174859 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.175037 (+ 178us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:11.175438 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.175929 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:11.176636 (+ 707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:11.176832 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:11.176915 (+ 83us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:11.177292 (+ 377us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:11.177417 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.177496 (+ 79us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.177543 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 20:00:11.183598 32706 raft_consensus.cc:1275] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 1 FOLLOWER]: Refusing update from remote peer 00f3090eef6145a781bf7aac0d54d0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:11.183601 373 raft_consensus.cc:1275] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 1 FOLLOWER]: Refusing update from remote peer 00f3090eef6145a781bf7aac0d54d0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:11.184278 442 consensus_queue.cc:1048] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d66c70cad951404db2ba2a9e06e37332" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 34771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:11.184594 447 consensus_queue.cc:1048] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28f61d059a094470be371f45114db575" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:11.192780 451 mvcc.cc:204] Tried to move back new op lower bound from 7280610554599972864 to 7280610554506641408. Current Snapshot: MvccSnapshot[applied={T|T < 7280610554599972864}]
I20260429 20:00:11.202991 32425 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:11.203168 32425 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:11.205798 32425 catalog_manager.cc:5958] T 00000000000000000000000000000000 P f7f6b669d1584ad7bda2a8f3679f389e: Sending DeleteTablet for 3 replicas of tablet d95eff924d8b4938bedb25c905081476
I20260429 20:00:11.206597 32551 tablet_service.cc:1558] Processing DeleteTablet for tablet d95eff924d8b4938bedb25c905081476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:11 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:60690
I20260429 20:00:11.206629 32686 tablet_service.cc:1558] Processing DeleteTablet for tablet d95eff924d8b4938bedb25c905081476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:11 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:50100
I20260429 20:00:11.206749 353 tablet_service.cc:1558] Processing DeleteTablet for tablet d95eff924d8b4938bedb25c905081476 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:11 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:33382
I20260429 20:00:11.207219 461 tablet_replica.cc:333] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: stopping tablet replica
I20260429 20:00:11.207474 462 tablet_replica.cc:333] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: stopping tablet replica
I20260429 20:00:11.207535 461 raft_consensus.cc:2243] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:11.207712 462 raft_consensus.cc:2243] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:11.207863 461 raft_consensus.cc:2272] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:11.207993 462 raft_consensus.cc:2272] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:11.208200 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 32489
I20260429 20:00:11.208243 460 tablet_replica.cc:333] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2: stopping tablet replica
I20260429 20:00:11.208494 460 raft_consensus.cc:2243] T d95eff924d8b4938bedb25c905081476 P 00f3090eef6145a781bf7aac0d54d0c2 [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:00:11.210196 461 ts_tablet_manager.cc:1916] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:11.210319 462 ts_tablet_manager.cc:1916] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:11.212438 462 ts_tablet_manager.cc:1929] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:11.212553 462 log.cc:1199] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/d95eff924d8b4938bedb25c905081476
I20260429 20:00:11.212937 462 ts_tablet_manager.cc:1950] T d95eff924d8b4938bedb25c905081476 P d66c70cad951404db2ba2a9e06e37332: Deleting consensus metadata
I20260429 20:00:11.213521 461 ts_tablet_manager.cc:1929] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:11.213625 461 log.cc:1199] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/d95eff924d8b4938bedb25c905081476
I20260429 20:00:11.213968 461 ts_tablet_manager.cc:1950] T d95eff924d8b4938bedb25c905081476 P 28f61d059a094470be371f45114db575: Deleting consensus metadata
I20260429 20:00:11.214648 32409 catalog_manager.cc:5002] TS d66c70cad951404db2ba2a9e06e37332 (127.27.17.195:34771): tablet d95eff924d8b4938bedb25c905081476 (table test-table [id=ace5052513a543cab4a32d946b2f2a90]) successfully deleted
I20260429 20:00:11.215941 32412 catalog_manager.cc:5002] TS 28f61d059a094470be371f45114db575 (127.27.17.194:35601): tablet d95eff924d8b4938bedb25c905081476 (table test-table [id=ace5052513a543cab4a32d946b2f2a90]) successfully deleted
W20260429 20:00:11.216579 32411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.17.193:35311 (error 108)
I20260429 20:00:11.216857 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 32624
I20260429 20:00:11.218464 432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.218181 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:35311 (local address 127.0.0.1:60710)
0429 20:00:11.218328 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.218384 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.193:35311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260429 20:00:11.218618 32411 catalog_manager.cc:4729] TS 00f3090eef6145a781bf7aac0d54d0c2 (127.27.17.193:35311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d95eff924d8b4938bedb25c905081476: Network error: Client connection negotiation failed: client connection to 127.27.17.193:35311: connect: Connection refused (error 111)
I20260429 20:00:11.223354 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 32759
I20260429 20:00:11.229609 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 32394
2026-04-29T20:00:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3382 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-04-29T20:00:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:11Z Disabled control of system clock
I20260429 20:00:11.261759 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:35943
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:35943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 20:00:11.375535 467 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:11.375895 467 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:11.375967 467 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:11.380690 467 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:11.380756 467 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:11.380774 467 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:11.380793 467 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:11.380811 467 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:11.384967 467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:35943
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:11.386041 467 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:11.387171 467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:11.392906 473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.392879 472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.392886 475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.392976 467 server_base.cc:1061] running on GCE node
I20260429 20:00:11.394322 467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:11.395367 467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:11.396562 467 hybrid_clock.cc:648] HybridClock initialized: now 1777492811396533 us; error 49 us; skew 500 ppm
I20260429 20:00:11.398391 467 webserver.cc:492] Webserver started at http://127.27.17.254:44667/ using document root <none> and password file <none>
I20260429 20:00:11.399130 467 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:11.399210 467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:11.399431 467 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:11.401103 467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "afe8989eec304bf69b5fddfc713f5dbd"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "0701e010066a6f641dccdfbf25ae384a"
server_key_iv: "1771bbb18e05a139b8d1d2d21eef59be"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.401602 467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "afe8989eec304bf69b5fddfc713f5dbd"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "0701e010066a6f641dccdfbf25ae384a"
server_key_iv: "1771bbb18e05a139b8d1d2d21eef59be"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.405052 467 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 20:00:11.407763 481 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.408886 467 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:11.409062 467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "afe8989eec304bf69b5fddfc713f5dbd"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "0701e010066a6f641dccdfbf25ae384a"
server_key_iv: "1771bbb18e05a139b8d1d2d21eef59be"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.409235 467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:11.414278 467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:11.414888 467 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:11.415159 467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:11.422798 467 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:35943
I20260429 20:00:11.422820 533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:35943 every 8 connection(s)
I20260429 20:00:11.423907 467 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:11.426695 534 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:11.428243 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 467
I20260429 20:00:11.428390 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:11.428651 27719 external_mini_cluster.cc:1468] Setting key 2d2bca3a2c40454e37e6f5950f841260
I20260429 20:00:11.432371 534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd: Bootstrap starting.
I20260429 20:00:11.434300 537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.430125 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33088 (local address 127.27.17.254:35943)
0429 20:00:11.430747 (+ 622us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.430758 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.430798 (+ 40us) server_negotiation.cc:408] Connection header received
0429 20:00:11.431486 (+ 688us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.431510 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.431879 (+ 369us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.432234 (+ 355us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:11.433016 (+ 782us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:11.433022 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:11.433034 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:11.433045 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:11.433097 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:11.433111 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:11.433271 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.433353 (+ 82us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.433568 (+ 215us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":123,"threads_started":1}
I20260429 20:00:11.435133 534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:11.435855 534 log.cc:826] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:11.437588 534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd: No bootstrap required, opened a new log
I20260429 20:00:11.440093 534 raft_consensus.cc:359] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe8989eec304bf69b5fddfc713f5dbd" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35943 } }
I20260429 20:00:11.440274 534 raft_consensus.cc:385] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:11.440348 534 raft_consensus.cc:740] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afe8989eec304bf69b5fddfc713f5dbd, State: Initialized, Role: FOLLOWER
I20260429 20:00:11.440789 534 consensus_queue.cc:260] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe8989eec304bf69b5fddfc713f5dbd" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35943 } }
I20260429 20:00:11.440932 534 raft_consensus.cc:399] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:11.440985 534 raft_consensus.cc:493] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:11.441066 534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:11.441964 534 raft_consensus.cc:515] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe8989eec304bf69b5fddfc713f5dbd" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35943 } }
I20260429 20:00:11.442308 534 leader_election.cc:304] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afe8989eec304bf69b5fddfc713f5dbd; no voters:
I20260429 20:00:11.442612 534 leader_election.cc:290] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:11.442791 539 raft_consensus.cc:2804] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:11.443059 539 raft_consensus.cc:697] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [term 1 LEADER]: Becoming Leader. State: Replica: afe8989eec304bf69b5fddfc713f5dbd, State: Running, Role: LEADER
I20260429 20:00:11.443640 539 consensus_queue.cc:237] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe8989eec304bf69b5fddfc713f5dbd" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35943 } }
I20260429 20:00:11.443914 534 sys_catalog.cc:565] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:11.445396 541 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader afe8989eec304bf69b5fddfc713f5dbd. Latest consensus state: current_term: 1 leader_uuid: "afe8989eec304bf69b5fddfc713f5dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe8989eec304bf69b5fddfc713f5dbd" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35943 } } }
I20260429 20:00:11.445395 540 sys_catalog.cc:455] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afe8989eec304bf69b5fddfc713f5dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afe8989eec304bf69b5fddfc713f5dbd" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 35943 } } }
I20260429 20:00:11.445672 540 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:11.445700 541 sys_catalog.cc:458] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:11.446182 549 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:11.449749 549 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:11.455680 549 catalog_manager.cc:1357] Generated new cluster ID: a1e6de2c786e46feb3b47b194379e96d
I20260429 20:00:11.455775 549 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:11.476052 549 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:11.476428 549 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:11.487682 549 catalog_manager.cc:6044] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd: Generated new TSK 0
I20260429 20:00:11.488449 549 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:00:11.492884 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35943
--builtin_ntp_servers=127.27.17.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260429 20:00:11.602530 558 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:11.602828 558 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:11.602963 558 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:11.606531 558 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:11.606650 558 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:11.606765 558 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:11.611153 558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:11.612375 558 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:11.613729 558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:11.621034 564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.621042 563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.621064 566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.621263 558 server_base.cc:1061] running on GCE node
I20260429 20:00:11.621716 558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:11.622296 558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:11.623476 558 hybrid_clock.cc:648] HybridClock initialized: now 1777492811623450 us; error 40 us; skew 500 ppm
I20260429 20:00:11.625540 558 webserver.cc:492] Webserver started at http://127.27.17.193:33579/ using document root <none> and password file <none>
I20260429 20:00:11.626137 558 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:11.626195 558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:11.626411 558 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:11.628258 558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "2c586ba06b914c78b62e333ee6cd3695"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "ade681af73b2467bb56cd3e2cf7e7dd1"
server_key_iv: "d371ec1372825ee03555a61f1568f8a1"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.628760 558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "2c586ba06b914c78b62e333ee6cd3695"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "ade681af73b2467bb56cd3e2cf7e7dd1"
server_key_iv: "d371ec1372825ee03555a61f1568f8a1"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.632266 558 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:11.634811 572 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.635931 558 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 20:00:11.636083 558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "2c586ba06b914c78b62e333ee6cd3695"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "ade681af73b2467bb56cd3e2cf7e7dd1"
server_key_iv: "d371ec1372825ee03555a61f1568f8a1"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.636212 558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:11.641443 558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:11.642098 558 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:11.642313 558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:11.642946 558 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:11.643998 558 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:11.644075 558 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.644153 558 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:11.644207 558 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.654637 558 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:39225
I20260429 20:00:11.654664 685 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:39225 every 8 connection(s)
I20260429 20:00:11.655670 558 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:11.658746 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 558
I20260429 20:00:11.658842 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:11.659128 27719 external_mini_cluster.cc:1468] Setting key 87ccab8559986c519f46f9c8e55457fb
I20260429 20:00:11.661222 537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.657418 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:42045 (local address 127.27.17.254:35943)
0429 20:00:11.657667 (+ 249us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.657671 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.658432 (+ 761us) server_negotiation.cc:408] Connection header received
0429 20:00:11.659450 (+ 1018us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.659454 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.659531 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.659655 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:11.660739 (+ 1084us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:11.660743 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:11.660746 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:11.660763 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:11.660782 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:11.660790 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:11.660878 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.661044 (+ 166us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.661090 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260429 20:00:11.661417 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35943
--builtin_ntp_servers=127.27.17.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260429 20:00:11.661938 688 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.657682 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35943 (local address 127.27.17.193:42045)
0429 20:00:11.658288 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.658324 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.659240 (+ 916us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.659814 (+ 574us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.659822 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.659884 (+ 62us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:11.660516 (+ 632us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:11.660522 (+ 6us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:11.660548 (+ 26us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:11.660596 (+ 48us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:11.660877 (+ 281us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:11.660885 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:11.660894 (+ 9us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.661091 (+ 197us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.661201 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":91,"threads_started":1}
I20260429 20:00:11.663131 686 heartbeater.cc:344] Connected to a master server at 127.27.17.254:35943
I20260429 20:00:11.663372 686 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:11.663779 686 heartbeater.cc:507] Master 127.27.17.254:35943 requested a full tablet report, sending...
I20260429 20:00:11.665158 498 ts_manager.cc:194] Registered new tserver with Master: 2c586ba06b914c78b62e333ee6cd3695 (127.27.17.193:39225)
W20260429 20:00:11.769474 689 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:11.769716 689 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:11.769769 689 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:11.773375 689 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:11.773460 689 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:11.773540 689 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:11.778074 689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:11.779266 689 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:11.780435 689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:11.787544 695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.787633 689 server_base.cc:1061] running on GCE node
W20260429 20:00:11.787606 694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.787541 697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.788143 689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:11.788806 689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:11.789994 689 hybrid_clock.cc:648] HybridClock initialized: now 1777492811789979 us; error 28 us; skew 500 ppm
I20260429 20:00:11.792296 689 webserver.cc:492] Webserver started at http://127.27.17.194:41449/ using document root <none> and password file <none>
I20260429 20:00:11.792915 689 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:11.792975 689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:11.793192 689 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:11.795181 689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "3ef5725b82834e07b0056b4b634e7a35"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "38e4ca77b0fe5ec907ced6577a9923bc"
server_key_iv: "b96f1dc99f93c0bdb5160a34cdc6acd5"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.795715 689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "3ef5725b82834e07b0056b4b634e7a35"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "38e4ca77b0fe5ec907ced6577a9923bc"
server_key_iv: "b96f1dc99f93c0bdb5160a34cdc6acd5"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.799212 689 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 20:00:11.801615 703 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.802714 689 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:11.802873 689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "3ef5725b82834e07b0056b4b634e7a35"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "38e4ca77b0fe5ec907ced6577a9923bc"
server_key_iv: "b96f1dc99f93c0bdb5160a34cdc6acd5"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.803049 689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:11.807871 689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:11.808475 689 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:11.808660 689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:11.809198 689 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:11.810168 689 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:11.810248 689 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.810487 689 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:11.810598 689 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.820935 689 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:36967
I20260429 20:00:11.820959 816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:36967 every 8 connection(s)
I20260429 20:00:11.821926 689 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:11.827378 537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.823794 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:33961 (local address 127.27.17.254:35943)
0429 20:00:11.823943 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.823947 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.824810 (+ 863us) server_negotiation.cc:408] Connection header received
0429 20:00:11.825594 (+ 784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.825597 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.825641 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.825726 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:11.826834 (+ 1108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:11.826837 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:11.826839 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:11.826850 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:11.826873 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:11.826880 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:11.827038 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.827191 (+ 153us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.827224 (+ 33us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 20:00:11.827812 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 689
I20260429 20:00:11.827947 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:11.828130 819 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.824072 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35943 (local address 127.27.17.194:33961)
0429 20:00:11.824655 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.824687 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.825357 (+ 670us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.825890 (+ 533us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.825898 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.825959 (+ 61us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:11.826593 (+ 634us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:11.826596 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:11.826621 (+ 25us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:11.826669 (+ 48us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:11.827034 (+ 365us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:11.827040 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:11.827045 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.827201 (+ 156us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.827376 (+ 175us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20260429 20:00:11.828220 27719 external_mini_cluster.cc:1468] Setting key 12cee05d9ad474e32de4fc7d50b30996
I20260429 20:00:11.829339 817 heartbeater.cc:344] Connected to a master server at 127.27.17.254:35943
I20260429 20:00:11.829623 817 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:11.830080 817 heartbeater.cc:507] Master 127.27.17.254:35943 requested a full tablet report, sending...
I20260429 20:00:11.830518 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35943
--builtin_ntp_servers=127.27.17.212:39679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260429 20:00:11.831374 497 ts_manager.cc:194] Registered new tserver with Master: 3ef5725b82834e07b0056b4b634e7a35 (127.27.17.194:36967)
W20260429 20:00:11.937844 820 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:11.938090 820 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:11.938143 820 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:11.941654 820 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:11.941718 820 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:11.941795 820 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:11.946007 820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:35943
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:11.947221 820 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:11.948283 820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:11.954998 828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.955008 825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:11.955066 826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:11.955279 820 server_base.cc:1061] running on GCE node
I20260429 20:00:11.955659 820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:11.956256 820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:11.957480 820 hybrid_clock.cc:648] HybridClock initialized: now 1777492811957461 us; error 45 us; skew 500 ppm
I20260429 20:00:11.959470 820 webserver.cc:492] Webserver started at http://127.27.17.195:41991/ using document root <none> and password file <none>
I20260429 20:00:11.960048 820 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:11.960132 820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:11.960376 820 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:11.962072 820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "c59bd5060e354fcfbc31d38082c2375d"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "1ce332b211baf7cf96aae273697b3eb7"
server_key_iv: "7b1334f80a1ece265a1c67683d0ba608"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.962584 820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "c59bd5060e354fcfbc31d38082c2375d"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "1ce332b211baf7cf96aae273697b3eb7"
server_key_iv: "7b1334f80a1ece265a1c67683d0ba608"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.966073 820 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 20:00:11.968649 834 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.969796 820 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 20:00:11.969959 820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "c59bd5060e354fcfbc31d38082c2375d"
format_stamp: "Formatted at 2026-04-29 20:00:11 on dist-test-slave-01nr"
server_key: "1ce332b211baf7cf96aae273697b3eb7"
server_key_iv: "7b1334f80a1ece265a1c67683d0ba608"
server_key_version: "encryptionkey@0"
I20260429 20:00:11.970093 820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:11.975129 820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:11.975724 820 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:11.975912 820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:11.976433 820 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:11.977391 820 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:11.977466 820 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.977536 820 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:11.977589 820 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:11.986963 820 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:46541
I20260429 20:00:11.986981 947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:46541 every 8 connection(s)
I20260429 20:00:11.988029 820 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:11.993618 537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.989820 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:56195 (local address 127.27.17.254:35943)
0429 20:00:11.989979 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:11.989983 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:11.990942 (+ 959us) server_negotiation.cc:408] Connection header received
0429 20:00:11.991780 (+ 838us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:11.991784 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:11.991833 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:11.991944 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:11.993039 (+ 1095us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:11.993043 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:11.993046 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:11.993059 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:11.993102 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:11.993111 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:11.993216 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:11.993408 (+ 192us) server_negotiation.cc:300] Negotiation successful
0429 20:00:11.993458 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 20:00:11.994268 950 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:11.990080 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:35943 (local address 127.27.17.195:56195)
0429 20:00:11.990741 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:11.990775 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:11.991570 (+ 795us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:11.992098 (+ 528us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:11.992106 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:11.992169 (+ 63us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:11.992788 (+ 619us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:11.992790 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:11.992818 (+ 28us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:11.992867 (+ 49us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:11.993208 (+ 341us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:11.993215 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:11.993221 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 20:00:11.993464 (+ 243us) client_negotiation.cc:241] Negotiation successful
0429 20:00:11.993580 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":98,"threads_started":1}
I20260429 20:00:11.995489 948 heartbeater.cc:344] Connected to a master server at 127.27.17.254:35943
I20260429 20:00:11.995738 948 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:11.996165 948 heartbeater.cc:507] Master 127.27.17.254:35943 requested a full tablet report, sending...
I20260429 20:00:11.996984 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 820
I20260429 20:00:11.997109 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:11.997336 497 ts_manager.cc:194] Registered new tserver with Master: c59bd5060e354fcfbc31d38082c2375d (127.27.17.195:46541)
I20260429 20:00:11.997371 27719 external_mini_cluster.cc:1468] Setting key 36c918983b90dde5bc80c8594351149d
I20260429 20:00:11.999805 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 20:00:12.003834 537 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.002015 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:33104 (local address 127.27.17.254:35943)
0429 20:00:12.002214 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.002218 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.002361 (+ 143us) server_negotiation.cc:408] Connection header received
0429 20:00:12.002506 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.002509 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.002559 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.002650 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.003365 (+ 715us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.003369 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.003372 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.003383 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.003412 (+ 29us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.003420 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.003516 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.003645 (+ 129us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.003680 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 20:00:12.006769 497 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:12.009436 497 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:12.023171 961 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.020253 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:36967 (local address 127.0.0.1:37542)
0429 20:00:12.020785 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.020801 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:12.020891 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:12.022003 (+ 1112us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:12.022009 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:12.022025 (+ 16us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:12.022458 (+ 433us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:12.022459 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:12.022470 (+ 11us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:12.022489 (+ 19us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:12.022784 (+ 295us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:12.022791 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:12.022796 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 20:00:12.022954 (+ 158us) client_negotiation.cc:241] Negotiation successful
0429 20:00:12.023000 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":108,"threads_started":1}
I20260429 20:00:12.023284 960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.019842 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44438 (local address 127.27.17.195:46541)
0429 20:00:12.020999 (+ 1157us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.021004 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.021023 (+ 19us) server_negotiation.cc:408] Connection header received
0429 20:00:12.021077 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.021081 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.021185 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.021710 (+ 525us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.022732 (+ 1022us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.022740 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.022752 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.022766 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.022801 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.022812 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.022947 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.023023 (+ 76us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.023097 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":158,"threads_started":1}
I20260429 20:00:12.023171 959 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.019667 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:46541 (local address 127.0.0.1:44438)
0429 20:00:12.020321 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.020362 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:12.020479 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:12.021379 (+ 900us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:12.021385 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:12.021445 (+ 60us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:12.022006 (+ 561us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:12.022008 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:12.022023 (+ 15us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:12.022066 (+ 43us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:12.022919 (+ 853us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:12.022925 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:12.022931 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 20:00:12.022999 (+ 68us) client_negotiation.cc:241] Negotiation successful
0429 20:00:12.023020 (+ 21us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":135,"threads_started":1}
I20260429 20:00:12.023679 963 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.020414 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.27.17.194:36967)
0429 20:00:12.020752 (+ 338us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.020759 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.020895 (+ 136us) server_negotiation.cc:408] Connection header received
0429 20:00:12.021010 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.021016 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.021186 (+ 170us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.021333 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.022616 (+ 1283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.022622 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.022632 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.022646 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.022684 (+ 38us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.022695 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.022828 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.023430 (+ 602us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.023502 (+ 72us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":128,"threads_started":1}
I20260429 20:00:12.024868 962 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.020434 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:39225 (local address 127.0.0.1:56590)
0429 20:00:12.021378 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.021390 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:12.021457 (+ 67us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:12.023622 (+ 2165us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:12.023625 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:12.023641 (+ 16us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:12.024197 (+ 556us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:12.024198 (+ 1us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:12.024208 (+ 10us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:12.024229 (+ 21us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:12.024629 (+ 400us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:12.024632 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:12.024635 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 20:00:12.024714 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 20:00:12.024743 (+ 29us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":88,"threads_started":1}
I20260429 20:00:12.024948 964 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.020829 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.27.17.193:39225)
0429 20:00:12.022394 (+ 1565us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.022402 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.022430 (+ 28us) server_negotiation.cc:408] Connection header received
0429 20:00:12.022509 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.022515 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.022678 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.022814 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.024378 (+ 1564us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.024384 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.024393 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.024404 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.024448 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.024461 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.024583 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.024739 (+ 156us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.024803 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1428,"thread_start_us":109,"threads_started":1}
I20260429 20:00:12.026436 751 tablet_service.cc:1511] Processing CreateTablet for tablet 9407adfc66cb4c1c8564213fee909307 (DEFAULT_TABLE table=test-table [id=b0211f9eaf904ba5bf29aef5bc8b9591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:12.026611 882 tablet_service.cc:1511] Processing CreateTablet for tablet 9407adfc66cb4c1c8564213fee909307 (DEFAULT_TABLE table=test-table [id=b0211f9eaf904ba5bf29aef5bc8b9591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:12.027514 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9407adfc66cb4c1c8564213fee909307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:12.027504 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9407adfc66cb4c1c8564213fee909307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:12.029906 620 tablet_service.cc:1511] Processing CreateTablet for tablet 9407adfc66cb4c1c8564213fee909307 (DEFAULT_TABLE table=test-table [id=b0211f9eaf904ba5bf29aef5bc8b9591]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:12.031160 620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9407adfc66cb4c1c8564213fee909307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:12.033939 965 tablet_bootstrap.cc:492] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Bootstrap starting.
I20260429 20:00:12.035254 966 tablet_bootstrap.cc:492] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Bootstrap starting.
I20260429 20:00:12.036653 965 tablet_bootstrap.cc:654] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:12.037600 965 log.cc:826] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:12.038038 966 tablet_bootstrap.cc:654] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:12.038841 968 tablet_bootstrap.cc:492] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: Bootstrap starting.
I20260429 20:00:12.038870 966 log.cc:826] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:12.039842 965 tablet_bootstrap.cc:492] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: No bootstrap required, opened a new log
I20260429 20:00:12.040060 965 ts_tablet_manager.cc:1403] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260429 20:00:12.040995 966 tablet_bootstrap.cc:492] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: No bootstrap required, opened a new log
I20260429 20:00:12.041181 968 tablet_bootstrap.cc:654] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:12.041204 966 ts_tablet_manager.cc:1403] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260429 20:00:12.042213 968 log.cc:826] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:12.043780 965 raft_consensus.cc:359] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.044019 965 raft_consensus.cc:385] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:12.044083 965 raft_consensus.cc:740] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59bd5060e354fcfbc31d38082c2375d, State: Initialized, Role: FOLLOWER
I20260429 20:00:12.044209 966 raft_consensus.cc:359] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.044409 966 raft_consensus.cc:385] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:12.044464 966 raft_consensus.cc:740] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ef5725b82834e07b0056b4b634e7a35, State: Initialized, Role: FOLLOWER
I20260429 20:00:12.044588 968 tablet_bootstrap.cc:492] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: No bootstrap required, opened a new log
I20260429 20:00:12.044582 965 consensus_queue.cc:260] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.044802 968 ts_tablet_manager.cc:1403] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 20:00:12.045032 966 consensus_queue.cc:260] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.045732 817 heartbeater.cc:499] Master 127.27.17.254:35943 was elected leader, sending a full tablet report...
I20260429 20:00:12.046150 966 ts_tablet_manager.cc:1434] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260429 20:00:12.046830 965 ts_tablet_manager.cc:1434] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260429 20:00:12.047219 948 heartbeater.cc:499] Master 127.27.17.254:35943 was elected leader, sending a full tablet report...
I20260429 20:00:12.048134 968 raft_consensus.cc:359] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.048389 968 raft_consensus.cc:385] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:12.048445 968 raft_consensus.cc:740] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c586ba06b914c78b62e333ee6cd3695, State: Initialized, Role: FOLLOWER
I20260429 20:00:12.048910 968 consensus_queue.cc:260] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.049742 686 heartbeater.cc:499] Master 127.27.17.254:35943 was elected leader, sending a full tablet report...
I20260429 20:00:12.050096 968 ts_tablet_manager.cc:1434] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
W20260429 20:00:12.072865 818 tablet.cc:2404] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 20:00:12.156828 687 tablet.cc:2404] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 20:00:12.239006 949 tablet.cc:2404] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 20:00:12.461966 973 raft_consensus.cc:493] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:12.462163 973 raft_consensus.cc:515] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.463464 973 leader_election.cc:290] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c59bd5060e354fcfbc31d38082c2375d (127.27.17.195:46541), 3ef5725b82834e07b0056b4b634e7a35 (127.27.17.194:36967)
I20260429 20:00:12.465718 960 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.463916 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:59819 (local address 127.27.17.195:46541)
0429 20:00:12.464180 (+ 264us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.464186 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.464202 (+ 16us) server_negotiation.cc:408] Connection header received
0429 20:00:12.464391 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.464394 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.464439 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.464559 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.465339 (+ 780us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.465342 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.465345 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.465357 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.465378 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.465386 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.465492 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.465553 (+ 61us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.465588 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260429 20:00:12.465739 974 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.463791 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:46541 (local address 127.27.17.193:59819)
0429 20:00:12.464116 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.464132 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:12.464236 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:12.464576 (+ 340us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:12.464579 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:12.464616 (+ 37us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:12.465164 (+ 548us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:12.465166 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:12.465175 (+ 9us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:12.465197 (+ 22us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:12.465477 (+ 280us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:12.465481 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:12.465484 (+ 3us) client_negotiation.cc:770] Sending connection context
0429 20:00:12.465561 (+ 77us) client_negotiation.cc:241] Negotiation successful
0429 20:00:12.465589 (+ 28us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
I20260429 20:00:12.466361 963 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.463910 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:37527 (local address 127.27.17.194:36967)
0429 20:00:12.464060 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.464064 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.464279 (+ 215us) server_negotiation.cc:408] Connection header received
0429 20:00:12.464439 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.464442 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.464483 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.464570 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.465960 (+ 1390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.465964 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.465967 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.465977 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.465995 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.466004 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.466059 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.466213 (+ 154us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.466248 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 20:00:12.466395 975 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.463791 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:36967 (local address 127.27.17.193:37527)
0429 20:00:12.464152 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.464171 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:12.464315 (+ 144us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:12.464589 (+ 274us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:12.464592 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:12.464634 (+ 42us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:12.465754 (+ 1120us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:00:12.465756 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:12.465764 (+ 8us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:00:12.465813 (+ 49us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:12.466123 (+ 310us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:12.466125 (+ 2us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:12.466127 (+ 2us) client_negotiation.cc:770] Sending connection context
0429 20:00:12.466221 (+ 94us) client_negotiation.cc:241] Negotiation successful
0429 20:00:12.466246 (+ 25us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":46,"thread_start_us":93,"threads_started":1}
I20260429 20:00:12.466549 902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9407adfc66cb4c1c8564213fee909307" candidate_uuid: "2c586ba06b914c78b62e333ee6cd3695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59bd5060e354fcfbc31d38082c2375d" is_pre_election: true
I20260429 20:00:12.466874 902 raft_consensus.cc:2468] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c586ba06b914c78b62e333ee6cd3695 in term 0.
I20260429 20:00:12.467051 771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9407adfc66cb4c1c8564213fee909307" candidate_uuid: "2c586ba06b914c78b62e333ee6cd3695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef5725b82834e07b0056b4b634e7a35" is_pre_election: true
I20260429 20:00:12.467294 771 raft_consensus.cc:2468] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c586ba06b914c78b62e333ee6cd3695 in term 0.
I20260429 20:00:12.467386 574 leader_election.cc:304] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c586ba06b914c78b62e333ee6cd3695, c59bd5060e354fcfbc31d38082c2375d; no voters:
I20260429 20:00:12.467682 973 raft_consensus.cc:2804] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:12.467757 973 raft_consensus.cc:493] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:12.467801 973 raft_consensus.cc:3060] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:12.468777 973 raft_consensus.cc:515] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.469115 973 leader_election.cc:290] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [CANDIDATE]: Term 1 election: Requested vote from peers c59bd5060e354fcfbc31d38082c2375d (127.27.17.195:46541), 3ef5725b82834e07b0056b4b634e7a35 (127.27.17.194:36967)
I20260429 20:00:12.469466 771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9407adfc66cb4c1c8564213fee909307" candidate_uuid: "2c586ba06b914c78b62e333ee6cd3695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ef5725b82834e07b0056b4b634e7a35"
I20260429 20:00:12.469586 771 raft_consensus.cc:3060] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:12.469691 902 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9407adfc66cb4c1c8564213fee909307" candidate_uuid: "2c586ba06b914c78b62e333ee6cd3695" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59bd5060e354fcfbc31d38082c2375d"
I20260429 20:00:12.469830 902 raft_consensus.cc:3060] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:12.470878 902 raft_consensus.cc:2468] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c586ba06b914c78b62e333ee6cd3695 in term 1.
I20260429 20:00:12.471156 771 raft_consensus.cc:2468] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c586ba06b914c78b62e333ee6cd3695 in term 1.
I20260429 20:00:12.471338 574 leader_election.cc:304] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c586ba06b914c78b62e333ee6cd3695, c59bd5060e354fcfbc31d38082c2375d; no voters:
I20260429 20:00:12.471583 973 raft_consensus.cc:2804] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:12.471902 973 raft_consensus.cc:697] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [term 1 LEADER]: Becoming Leader. State: Replica: 2c586ba06b914c78b62e333ee6cd3695, State: Running, Role: LEADER
I20260429 20:00:12.472262 973 consensus_queue.cc:237] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } }
I20260429 20:00:12.475908 497 catalog_manager.cc:5671] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c586ba06b914c78b62e333ee6cd3695 (127.27.17.193). New cstate: current_term: 1 leader_uuid: "2c586ba06b914c78b62e333ee6cd3695" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c586ba06b914c78b62e333ee6cd3695" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:12.591483 980 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.589238 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.27.17.193:39225)
0429 20:00:12.589601 (+ 363us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.589605 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.589622 (+ 17us) server_negotiation.cc:408] Connection header received
0429 20:00:12.589828 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.589832 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.589905 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.589997 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.590959 (+ 962us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.590962 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.590965 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.590975 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.590997 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.591005 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.591091 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.591232 (+ 141us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.591264 (+ 32us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":148,"threads_started":1}
I20260429 20:00:12.597364 902 raft_consensus.cc:1275] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 1 FOLLOWER]: Refusing update from remote peer 2c586ba06b914c78b62e333ee6cd3695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:12.597384 771 raft_consensus.cc:1275] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 1 FOLLOWER]: Refusing update from remote peer 2c586ba06b914c78b62e333ee6cd3695: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:12.598101 976 consensus_queue.cc:1048] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59bd5060e354fcfbc31d38082c2375d" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:12.598320 973 consensus_queue.cc:1048] T 9407adfc66cb4c1c8564213fee909307 P 2c586ba06b914c78b62e333ee6cd3695 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ef5725b82834e07b0056b4b634e7a35" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 36967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:12.607201 984 mvcc.cc:204] Tried to move back new op lower bound from 7280610560391041024 to 7280610559890038784. Current Snapshot: MvccSnapshot[applied={T|T < 7280610560391041024}]
I20260429 20:00:12.616639 496 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:12.616838 496 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:12.619925 496 catalog_manager.cc:5958] T 00000000000000000000000000000000 P afe8989eec304bf69b5fddfc713f5dbd: Sending DeleteTablet for 3 replicas of tablet 9407adfc66cb4c1c8564213fee909307
I20260429 20:00:12.620913 620 tablet_service.cc:1558] Processing DeleteTablet for tablet 9407adfc66cb4c1c8564213fee909307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:12 UTC) from {username='slave'} at 127.0.0.1:56590
I20260429 20:00:12.620888 882 tablet_service.cc:1558] Processing DeleteTablet for tablet 9407adfc66cb4c1c8564213fee909307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:12 UTC) from {username='slave'} at 127.0.0.1:44438
I20260429 20:00:12.621425 751 tablet_service.cc:1558] Processing DeleteTablet for tablet 9407adfc66cb4c1c8564213fee909307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:12 UTC) from {username='slave'} at 127.0.0.1:37542
I20260429 20:00:12.621680 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 558
I20260429 20:00:12.622807 997 tablet_replica.cc:333] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: stopping tablet replica
I20260429 20:00:12.623152 997 raft_consensus.cc:2243] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:12.623518 997 raft_consensus.cc:2272] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:12.623945 996 tablet_replica.cc:333] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: stopping tablet replica
I20260429 20:00:12.624219 996 raft_consensus.cc:2243] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:12.624531 996 raft_consensus.cc:2272] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:12.624994 997 ts_tablet_manager.cc:1916] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:12.625787 996 ts_tablet_manager.cc:1916] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:12.627342 997 ts_tablet_manager.cc:1929] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:12.627452 997 log.cc:1199] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/9407adfc66cb4c1c8564213fee909307
I20260429 20:00:12.627729 997 ts_tablet_manager.cc:1950] T 9407adfc66cb4c1c8564213fee909307 P 3ef5725b82834e07b0056b4b634e7a35: Deleting consensus metadata
I20260429 20:00:12.629088 996 ts_tablet_manager.cc:1929] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:12.629108 482 catalog_manager.cc:5002] TS 3ef5725b82834e07b0056b4b634e7a35 (127.27.17.194:36967): tablet 9407adfc66cb4c1c8564213fee909307 (table test-table [id=b0211f9eaf904ba5bf29aef5bc8b9591]) successfully deleted
I20260429 20:00:12.629191 996 log.cc:1199] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/9407adfc66cb4c1c8564213fee909307
I20260429 20:00:12.629516 996 ts_tablet_manager.cc:1950] T 9407adfc66cb4c1c8564213fee909307 P c59bd5060e354fcfbc31d38082c2375d: Deleting consensus metadata
W20260429 20:00:12.630236 483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.17.193:39225 (error 108)
I20260429 20:00:12.631225 483 catalog_manager.cc:5002] TS c59bd5060e354fcfbc31d38082c2375d (127.27.17.195:46541): tablet 9407adfc66cb4c1c8564213fee909307 (table test-table [id=b0211f9eaf904ba5bf29aef5bc8b9591]) successfully deleted
I20260429 20:00:12.631460 999 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.631044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:39225 (local address 127.0.0.1:56616)
0429 20:00:12.631242 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.631356 (+ 114us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.193:39225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":48,"threads_started":1}
I20260429 20:00:12.631646 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 689
W20260429 20:00:12.631636 483 catalog_manager.cc:4729] TS 2c586ba06b914c78b62e333ee6cd3695 (127.27.17.193:39225): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9407adfc66cb4c1c8564213fee909307: Network error: Client connection negotiation failed: client connection to 127.27.17.193:39225: connect: Connection refused (error 111)
I20260429 20:00:12.638373 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 820
I20260429 20:00:12.645171 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 467
2026-04-29T20:00:12Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1414 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260429 20:00:12.668071 27719 mini_oidc.cc:150] Starting JWKS server
I20260429 20:00:12.669482 27719 webserver.cc:492] Webserver started at https://127.0.0.1:39815/ using document root <none> and password file <none>
I20260429 20:00:12.669643 27719 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 20:00:12.669885 27719 webserver.cc:492] Webserver started at http://127.0.0.1:35243/ using document root <none> and password file <none>
2026-04-29T20:00:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:12Z Disabled control of system clock
I20260429 20:00:12.678486 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45973
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:45973
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39815/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/testchainca.cert with env {}
W20260429 20:00:12.786031 1005 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:12.786314 1005 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:12.786417 1005 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:12.789959 1005 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:12.790050 1005 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:12.790067 1005 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:12.790087 1005 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:12.790105 1005 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 20:00:12.790126 1005 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 20:00:12.790373 1005 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:39815/jwks.json
I20260429 20:00:12.795291 1005 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:45973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39815/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:12.796482 1005 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:12.797384 1005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:12.803581 1011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:12.803584 1010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:12.803707 1005 server_base.cc:1061] running on GCE node
W20260429 20:00:12.803833 1013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:12.804330 1005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:12.805403 1005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:12.806568 1005 hybrid_clock.cc:648] HybridClock initialized: now 1777492812806528 us; error 54 us; skew 500 ppm
I20260429 20:00:12.808455 1005 webserver.cc:492] Webserver started at http://127.27.17.254:46195/ using document root <none> and password file <none>
I20260429 20:00:12.808986 1005 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:12.809037 1005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:12.809186 1005 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:12.810878 1005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "c699d295b9374e77ab21496c3b24fdf9"
format_stamp: "Formatted at 2026-04-29 20:00:12 on dist-test-slave-01nr"
server_key: "1b14403a7dfc7af1b404524bf1f7ac46"
server_key_iv: "f65c0f4bc0e1441eef1c61a8ceab5507"
server_key_version: "encryptionkey@0"
I20260429 20:00:12.811432 1005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "c699d295b9374e77ab21496c3b24fdf9"
format_stamp: "Formatted at 2026-04-29 20:00:12 on dist-test-slave-01nr"
server_key: "1b14403a7dfc7af1b404524bf1f7ac46"
server_key_iv: "f65c0f4bc0e1441eef1c61a8ceab5507"
server_key_version: "encryptionkey@0"
I20260429 20:00:12.815006 1005 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260429 20:00:12.817235 1019 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:12.818427 1005 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:12.818543 1005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "c699d295b9374e77ab21496c3b24fdf9"
format_stamp: "Formatted at 2026-04-29 20:00:12 on dist-test-slave-01nr"
server_key: "1b14403a7dfc7af1b404524bf1f7ac46"
server_key_iv: "f65c0f4bc0e1441eef1c61a8ceab5507"
server_key_version: "encryptionkey@0"
I20260429 20:00:12.818650 1005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:12.829074 1005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:12.829751 1005 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:12.829913 1005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:12.838447 1005 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:45973
I20260429 20:00:12.838434 1071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:45973 every 8 connection(s)
I20260429 20:00:12.839692 1005 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:12.843427 1072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:12.844575 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1005
I20260429 20:00:12.844692 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:12.845033 27719 external_mini_cluster.cc:1468] Setting key 313e6a1057d650db9e2e7861dbdd866c
I20260429 20:00:12.850951 1072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9: Bootstrap starting.
I20260429 20:00:12.853998 1072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:12.854867 1075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.846575 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.27.17.254:45973)
0429 20:00:12.847331 (+ 756us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.847346 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.847392 (+ 46us) server_negotiation.cc:408] Connection header received
0429 20:00:12.848118 (+ 726us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.848145 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.848629 (+ 484us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.849018 (+ 389us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:12.849587 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.850735 (+ 1148us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:12.851729 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.852012 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:12.852634 (+ 622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:12.852688 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:12.852705 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:12.853214 (+ 509us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:12.853262 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:12.853281 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:12.853404 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.853881 (+ 477us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.854058 (+ 177us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":137,"threads_started":1}
I20260429 20:00:12.854979 1072 log.cc:826] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:12.857436 1072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9: No bootstrap required, opened a new log
I20260429 20:00:12.859088 27719 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 20:00:12.861442 1072 raft_consensus.cc:359] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c699d295b9374e77ab21496c3b24fdf9" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45973 } }
I20260429 20:00:12.861683 1072 raft_consensus.cc:385] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:12.861723 1072 raft_consensus.cc:740] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c699d295b9374e77ab21496c3b24fdf9, State: Initialized, Role: FOLLOWER
I20260429 20:00:12.862085 1072 consensus_queue.cc:260] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c699d295b9374e77ab21496c3b24fdf9" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45973 } }
I20260429 20:00:12.862192 1072 raft_consensus.cc:399] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:12.862232 1072 raft_consensus.cc:493] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:12.862298 1072 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:12.863467 1072 raft_consensus.cc:515] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c699d295b9374e77ab21496c3b24fdf9" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45973 } }
I20260429 20:00:12.863816 1072 leader_election.cc:304] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c699d295b9374e77ab21496c3b24fdf9; no voters:
I20260429 20:00:12.864089 1072 leader_election.cc:290] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:12.864383 1077 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:12.864701 1077 raft_consensus.cc:697] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [term 1 LEADER]: Becoming Leader. State: Replica: c699d295b9374e77ab21496c3b24fdf9, State: Running, Role: LEADER
I20260429 20:00:12.865105 1077 consensus_queue.cc:237] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c699d295b9374e77ab21496c3b24fdf9" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45973 } }
I20260429 20:00:12.865366 1072 sys_catalog.cc:565] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:12.867103 1078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c699d295b9374e77ab21496c3b24fdf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c699d295b9374e77ab21496c3b24fdf9" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45973 } } }
I20260429 20:00:12.867269 1078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:12.867283 1079 sys_catalog.cc:455] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c699d295b9374e77ab21496c3b24fdf9. Latest consensus state: current_term: 1 leader_uuid: "c699d295b9374e77ab21496c3b24fdf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c699d295b9374e77ab21496c3b24fdf9" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45973 } } }
I20260429 20:00:12.867509 1079 sys_catalog.cc:458] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:12.867712 1090 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:12.871120 1090 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:12.877607 1090 catalog_manager.cc:1357] Generated new cluster ID: e0e605951feb4720be7507a64f4ca40b
I20260429 20:00:12.877816 1090 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:12.884099 1090 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:12.885399 1090 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:12.903357 1090 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c699d295b9374e77ab21496c3b24fdf9: Generated new TSK 0
I20260429 20:00:12.904198 1090 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:00:12.950333 1075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.924921 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.27.17.254:45973)
0429 20:00:12.925072 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.925077 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.925238 (+ 161us) server_negotiation.cc:408] Connection header received
0429 20:00:12.925453 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.925457 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.925529 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.925646 (+ 117us) server_negotiation.cc:227] Negotiated authn=JWT
0429 20:00:12.926086 (+ 440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.926984 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:12.927805 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.928022 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:12.928121 (+ 99us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 20:00:12.949726 (+ 21605us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
0429 20:00:12.949867 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:12.950093 (+ 226us) server_negotiation.cc:300] Negotiation successful
0429 20:00:12.950156 (+ 63us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"thread_start_us":173,"threads_started":1}
I20260429 20:00:12.963483 1075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:12.959204 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.27.17.254:45973)
0429 20:00:12.959345 (+ 141us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:12.959350 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:12.959500 (+ 150us) server_negotiation.cc:408] Connection header received
0429 20:00:12.959605 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:12.959608 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:12.959663 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:12.959751 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
0429 20:00:12.960089 (+ 338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.960884 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:12.961569 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.961822 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:12.961897 (+ 75us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 20:00:12.963127 (+ 1230us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0429 20:00:12.963347 (+ 220us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37096: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":57}
W20260429 20:00:12.963569 1075 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37096: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20260429 20:00:12.964259 1108 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:12.959108 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45973 (local address 127.0.0.1:37096)
0429 20:00:12.959403 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:12.959417 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:12.959522 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:12.959768 (+ 246us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:12.959771 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:12.959777 (+ 6us) client_negotiation.cc:190] Negotiated authn=JWT
0429 20:00:12.959982 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:12.959988 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.961026 (+ 1038us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:12.961030 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:12.961445 (+ 415us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:12.961453 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:12.961546 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:12.961567 (+ 21us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 20:00:12.963359 (+ 1792us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
0429 20:00:12.963522 (+ 163us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.17.254:45973: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":113,"threads_started":1}
I20260429 20:00:27.973507 1114 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:27.969980 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.27.17.254:45973)
0429 20:00:27.970309 (+ 329us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:27.970314 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:27.970333 (+ 19us) server_negotiation.cc:408] Connection header received
0429 20:00:27.970462 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:27.970466 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:27.970516 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:27.970615 (+ 99us) server_negotiation.cc:227] Negotiated authn=JWT
0429 20:00:27.971039 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:27.971595 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:27.972293 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:27.972486 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:27.972531 (+ 45us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 20:00:27.973155 (+ 624us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0429 20:00:27.973351 (+ 196us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39450: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
W20260429 20:00:27.973580 1113 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:27.969805 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45973 (local address 127.0.0.1:39450)
0429 20:00:27.970160 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:27.970178 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:27.970327 (+ 149us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:27.970632 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:27.970636 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:27.970645 (+ 9us) client_negotiation.cc:190] Negotiated authn=JWT
0429 20:00:27.970885 (+ 240us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:27.970916 (+ 31us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:27.971720 (+ 804us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:27.971725 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:27.972169 (+ 444us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:27.972178 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:27.972306 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:27.972334 (+ 28us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 20:00:27.973292 (+ 958us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
0429 20:00:27.973429 (+ 137us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.17.254:45973: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":153,"threads_started":1}
W20260429 20:00:27.973593 1114 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39450: JWT verification failed: token expired
W20260429 20:00:27.976519 1114 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:27.975648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.27.17.254:45973)
0429 20:00:27.975807 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:27.975811 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:27.975908 (+ 97us) server_negotiation.cc:408] Connection header received
0429 20:00:27.976050 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:27.976055 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:27.976108 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:27.976192 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:27.976389 (+ 197us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39464: BlockingRecv error: recv got EOF from 127.0.0.1:39464 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260429 20:00:27.980439 1124 client_negotiation.cc:376] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.27.17.254:45973
W20260429 20:00:27.981103 1114 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:27.980072 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.27.17.254:45973)
0429 20:00:27.980225 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:27.980230 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:27.980361 (+ 131us) server_negotiation.cc:408] Connection header received
0429 20:00:27.980687 (+ 326us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:27.980690 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:27.980733 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:27.980821 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:27.980992 (+ 171us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39478: BlockingRecv error: recv got EOF from 127.0.0.1:39478 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 20:00:27.981284 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1005
2026-04-29T20:00:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15354 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260429 20:00:28.022197 27719 mini_oidc.cc:150] Starting JWKS server
I20260429 20:00:28.023200 27719 webserver.cc:492] Webserver started at https://127.0.0.1:38003/ using document root <none> and password file <none>
I20260429 20:00:28.023316 27719 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 20:00:28.023566 27719 webserver.cc:492] Webserver started at http://127.0.0.1:46151/ using document root <none> and password file <none>
2026-04-29T20:00:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:28Z Disabled control of system clock
I20260429 20:00:28.032279 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:40453
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:40453
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38003/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/testchainca.cert with env {}
W20260429 20:00:28.140555 1130 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:28.140810 1130 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:28.140865 1130 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:28.144866 1130 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:28.144958 1130 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:28.144975 1130 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:28.144995 1130 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:28.145013 1130 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 20:00:28.145032 1130 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 20:00:28.145047 1130 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:38003/jwks.json
I20260429 20:00:28.150102 1130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:40453
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:40453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38003/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:28.151350 1130 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:28.152181 1130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:28.158648 1136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:28.158648 1138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:28.158710 1130 server_base.cc:1061] running on GCE node
W20260429 20:00:28.158653 1135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:28.159433 1130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:28.160360 1130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:28.161535 1130 hybrid_clock.cc:648] HybridClock initialized: now 1777492828161510 us; error 40 us; skew 500 ppm
I20260429 20:00:28.163513 1130 webserver.cc:492] Webserver started at http://127.27.17.254:40801/ using document root <none> and password file <none>
I20260429 20:00:28.164081 1130 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:28.164139 1130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:28.164342 1130 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:28.166023 1130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "6a1edd6397f548b88df6fffc66f66a8a"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "de3804be0b9e6d417d81216910453e26"
server_key_iv: "8af2cf28a793eb8dd3d6e69c39102980"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.166522 1130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "6a1edd6397f548b88df6fffc66f66a8a"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "de3804be0b9e6d417d81216910453e26"
server_key_iv: "8af2cf28a793eb8dd3d6e69c39102980"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.170017 1130 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 20:00:28.172528 1144 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:28.173601 1130 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 20:00:28.173727 1130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "6a1edd6397f548b88df6fffc66f66a8a"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "de3804be0b9e6d417d81216910453e26"
server_key_iv: "8af2cf28a793eb8dd3d6e69c39102980"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.173838 1130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:28.191829 1130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:28.192520 1130 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:28.192672 1130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:28.200117 1130 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:40453
I20260429 20:00:28.200114 1196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:40453 every 8 connection(s)
I20260429 20:00:28.201171 1130 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:28.204001 1197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:28.208230 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1130
I20260429 20:00:28.208359 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:28.208600 27719 external_mini_cluster.cc:1468] Setting key f4122e9421b4476b57ab0b433a6f140c
I20260429 20:00:28.210068 1197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a: Bootstrap starting.
I20260429 20:00:28.212249 1197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:28.213063 1197 log.cc:826] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:28.215071 1197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a: No bootstrap required, opened a new log
I20260429 20:00:28.216403 1200 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:28.209975 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54964 (local address 127.27.17.254:40453)
0429 20:00:28.210601 (+ 626us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:28.210610 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:28.210642 (+ 32us) server_negotiation.cc:408] Connection header received
0429 20:00:28.211233 (+ 591us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:28.211252 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:28.211600 (+ 348us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:28.211913 (+ 313us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:28.212449 (+ 536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.213224 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.213889 (+ 665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.214186 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.214826 (+ 640us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:28.214851 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:28.214864 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:28.215366 (+ 502us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:28.215402 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:28.215416 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:28.215527 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:28.215696 (+ 169us) server_negotiation.cc:300] Negotiation successful
0429 20:00:28.215845 (+ 149us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":141,"threads_started":1}
I20260429 20:00:28.217748 1197 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1edd6397f548b88df6fffc66f66a8a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 40453 } }
I20260429 20:00:28.217999 1197 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:28.218077 1197 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1edd6397f548b88df6fffc66f66a8a, State: Initialized, Role: FOLLOWER
I20260429 20:00:28.218492 1197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1edd6397f548b88df6fffc66f66a8a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 40453 } }
I20260429 20:00:28.218652 1197 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:28.218721 1197 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:28.218823 1197 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:28.218997 27719 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 20:00:28.219978 1197 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1edd6397f548b88df6fffc66f66a8a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 40453 } }
I20260429 20:00:28.220347 1197 leader_election.cc:304] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1edd6397f548b88df6fffc66f66a8a; no voters:
I20260429 20:00:28.220681 1197 leader_election.cc:290] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:28.220809 1202 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:28.221065 1202 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [term 1 LEADER]: Becoming Leader. State: Replica: 6a1edd6397f548b88df6fffc66f66a8a, State: Running, Role: LEADER
I20260429 20:00:28.221388 1202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1edd6397f548b88df6fffc66f66a8a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 40453 } }
I20260429 20:00:28.221887 1197 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:28.223057 1204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1edd6397f548b88df6fffc66f66a8a. Latest consensus state: current_term: 1 leader_uuid: "6a1edd6397f548b88df6fffc66f66a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1edd6397f548b88df6fffc66f66a8a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 40453 } } }
I20260429 20:00:28.223081 1203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1edd6397f548b88df6fffc66f66a8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1edd6397f548b88df6fffc66f66a8a" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 40453 } } }
I20260429 20:00:28.223169 1204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:28.223199 1203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a [sys.catalog]: This master's current role is: LEADER
W20260429 20:00:28.227269 1218 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:00:28.227365 1218 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:00:28.227447 1211 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:28.228223 1211 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:28.233306 1211 catalog_manager.cc:1357] Generated new cluster ID: 1dc1edd5c0f34c61b62ad68c94e22547
I20260429 20:00:28.233389 1211 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:28.243031 1211 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:28.243927 1211 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:28.260871 1211 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 6a1edd6397f548b88df6fffc66f66a8a: Generated new TSK 0
I20260429 20:00:28.261600 1211 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260429 20:00:28.290251 1227 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260429 20:00:28.290354 1227 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260429 20:00:28.291121 1200 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:28.273995 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:54970 (local address 127.27.17.254:40453)
0429 20:00:28.274143 (+ 148us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:28.274147 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:28.274294 (+ 147us) server_negotiation.cc:408] Connection header received
0429 20:00:28.274436 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:28.274439 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:28.274501 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:28.274598 (+ 97us) server_negotiation.cc:227] Negotiated authn=JWT
0429 20:00:28.275195 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.275781 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.276746 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.276929 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.277019 (+ 90us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 20:00:28.290696 (+ 13677us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38003/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 20:00:28.290991 (+ 295us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54970: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38003/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":61}
W20260429 20:00:28.291215 1200 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54970: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38003/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260429 20:00:28.291342 1226 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:28.273813 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:40453 (local address 127.0.0.1:54970)
0429 20:00:28.274163 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:28.274179 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:28.274310 (+ 131us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:28.274624 (+ 314us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:28.274628 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:28.274636 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0429 20:00:28.275034 (+ 398us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:28.275047 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.275919 (+ 872us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.275924 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:28.276602 (+ 678us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:28.276614 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.276737 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.276767 (+ 30us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 20:00:28.290943 (+ 14176us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38003/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 20:00:28.291143 (+ 200us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.17.254:40453: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38003/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":155,"threads_started":1}
I20260429 20:00:28.292455 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1130
2026-04-29T20:00:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (310 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260429 20:00:28.332684 27719 mini_oidc.cc:150] Starting JWKS server
I20260429 20:00:28.333587 27719 webserver.cc:492] Webserver started at https://127.0.0.1:38069/ using document root <none> and password file <none>
I20260429 20:00:28.333670 27719 mini_oidc.cc:186] Starting OIDC Discovery server
I20260429 20:00:28.333843 27719 webserver.cc:492] Webserver started at http://127.0.0.1:46647/ using document root <none> and password file <none>
2026-04-29T20:00:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:28Z Disabled control of system clock
I20260429 20:00:28.342029 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:38223
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:43573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:38223
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260429 20:00:28.451821 1233 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:28.452092 1233 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:28.452143 1233 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:28.455739 1233 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:28.455816 1233 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:28.455833 1233 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:28.455852 1233 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:28.455871 1233 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 20:00:28.455946 1233 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 20:00:28.455977 1233 flags.cc:432] Enabled experimental flag: --jwks_url=https://localhost:38069/jwks.json
I20260429 20:00:28.460256 1233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:43573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:38223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:38223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:28.461364 1233 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:28.462473 1233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:28.468371 1238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:28.468349 1241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:28.468338 1239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:28.468631 1233 server_base.cc:1061] running on GCE node
I20260429 20:00:28.469115 1233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:28.470044 1233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:28.471200 1233 hybrid_clock.cc:648] HybridClock initialized: now 1777492828471173 us; error 39 us; skew 500 ppm
I20260429 20:00:28.473230 1233 webserver.cc:492] Webserver started at http://127.27.17.254:40125/ using document root <none> and password file <none>
I20260429 20:00:28.473834 1233 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:28.473892 1233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:28.474103 1233 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:28.475966 1233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "958f03016fd54f0698d8f74b327e8a75"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "40618ed052dcbd74527e13589187b6a1"
server_key_iv: "c14229da47279964b663909bc4c99c7b"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.476506 1233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "958f03016fd54f0698d8f74b327e8a75"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "40618ed052dcbd74527e13589187b6a1"
server_key_iv: "c14229da47279964b663909bc4c99c7b"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.480221 1233 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:28.482798 1247 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:28.483980 1233 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:28.484143 1233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "958f03016fd54f0698d8f74b327e8a75"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "40618ed052dcbd74527e13589187b6a1"
server_key_iv: "c14229da47279964b663909bc4c99c7b"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.484301 1233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:28.499712 1233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:28.500391 1233 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:28.500579 1233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:28.507961 1233 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:38223
I20260429 20:00:28.507967 1299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:38223 every 8 connection(s)
I20260429 20:00:28.509024 1233 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:28.512082 1300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:28.517588 1300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75: Bootstrap starting.
I20260429 20:00:28.517694 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1233
I20260429 20:00:28.517845 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:28.518077 27719 external_mini_cluster.cc:1468] Setting key 6a4ba4fa78f6975e78543972bbad9c8b
I20260429 20:00:28.520211 1300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:28.520956 1300 log.cc:826] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:28.523031 1300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75: No bootstrap required, opened a new log
I20260429 20:00:28.525790 1303 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:28.519231 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.27.17.254:38223)
0429 20:00:28.519790 (+ 559us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:28.519799 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:28.519834 (+ 35us) server_negotiation.cc:408] Connection header received
0429 20:00:28.520448 (+ 614us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:28.520468 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:28.520812 (+ 344us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:28.521108 (+ 296us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:28.521642 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.522437 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.523360 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.523562 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.524136 (+ 574us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:28.524158 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:28.524169 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:28.524581 (+ 412us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:28.524618 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:28.524630 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:28.524715 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:28.524923 (+ 208us) server_negotiation.cc:300] Negotiation successful
0429 20:00:28.525118 (+ 195us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":134,"threads_started":1}
I20260429 20:00:28.526008 1300 raft_consensus.cc:359] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f03016fd54f0698d8f74b327e8a75" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38223 } }
I20260429 20:00:28.526207 1300 raft_consensus.cc:385] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:28.526268 1300 raft_consensus.cc:740] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958f03016fd54f0698d8f74b327e8a75, State: Initialized, Role: FOLLOWER
I20260429 20:00:28.526682 1300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f03016fd54f0698d8f74b327e8a75" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38223 } }
I20260429 20:00:28.526808 1300 raft_consensus.cc:399] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:28.526924 1300 raft_consensus.cc:493] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:28.527006 1300 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:28.527978 1300 raft_consensus.cc:515] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f03016fd54f0698d8f74b327e8a75" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38223 } }
I20260429 20:00:28.528311 1300 leader_election.cc:304] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 958f03016fd54f0698d8f74b327e8a75; no voters:
I20260429 20:00:28.528636 1300 leader_election.cc:290] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:28.528714 1305 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:28.528749 27719 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 20:00:28.528877 1305 raft_consensus.cc:697] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [term 1 LEADER]: Becoming Leader. State: Replica: 958f03016fd54f0698d8f74b327e8a75, State: Running, Role: LEADER
I20260429 20:00:28.529209 1305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f03016fd54f0698d8f74b327e8a75" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38223 } }
I20260429 20:00:28.529837 1300 sys_catalog.cc:565] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:28.530359 1307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 958f03016fd54f0698d8f74b327e8a75. Latest consensus state: current_term: 1 leader_uuid: "958f03016fd54f0698d8f74b327e8a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f03016fd54f0698d8f74b327e8a75" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38223 } } }
I20260429 20:00:28.530480 1307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:28.530812 1306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "958f03016fd54f0698d8f74b327e8a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "958f03016fd54f0698d8f74b327e8a75" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 38223 } } }
I20260429 20:00:28.530964 1306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:28.531188 1309 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:28.534273 1309 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:28.539880 1309 catalog_manager.cc:1357] Generated new cluster ID: 5b00434612424afabda8293dcd42ae26
I20260429 20:00:28.539983 1309 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:28.571765 1309 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:28.572675 1309 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:28.582939 1309 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 958f03016fd54f0698d8f74b327e8a75: Generated new TSK 0
I20260429 20:00:28.583644 1309 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20260429 20:00:28.621210 1330 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260429 20:00:28.621311 1330 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260429 20:00:28.623008 1303 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:28.597887 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.27.17.254:38223)
0429 20:00:28.598023 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:28.598027 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:28.598153 (+ 126us) server_negotiation.cc:408] Connection header received
0429 20:00:28.598330 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:28.598337 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:28.598395 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:28.598504 (+ 109us) server_negotiation.cc:227] Negotiated authn=JWT
0429 20:00:28.599027 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.599623 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.600585 (+ 962us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.600749 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.600833 (+ 84us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 20:00:28.622500 (+ 21667us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 20:00:28.622825 (+ 325us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48270: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260429 20:00:28.623090 1303 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48270: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260429 20:00:28.623185 1329 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:28.597708 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:38223 (local address 127.0.0.1:48270)
0429 20:00:28.598020 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:28.598037 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:28.598158 (+ 121us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:28.598523 (+ 365us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:28.598528 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:28.598536 (+ 8us) client_negotiation.cc:190] Negotiated authn=JWT
0429 20:00:28.598828 (+ 292us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:28.598838 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.599763 (+ 925us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.599768 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:28.600452 (+ 684us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:28.600464 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.600606 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.600632 (+ 26us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 20:00:28.622739 (+ 22107us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0429 20:00:28.622968 (+ 229us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.17.254:38223: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":127,"threads_started":1}
I20260429 20:00:28.623994 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1233
2026-04-29T20:00:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (332 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-04-29T20:00:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:28Z Disabled control of system clock
I20260429 20:00:28.673324 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:36267
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:39143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:36267
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/testchainca.cert with env {}
W20260429 20:00:28.784811 1334 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:28.785077 1334 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:28.785133 1334 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:28.788810 1334 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:28.788889 1334 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:28.788906 1334 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:28.788925 1334 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:28.788944 1334 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260429 20:00:28.788961 1334 flags.cc:432] Enabled experimental flag: --enable_jwt_token_auth=true
W20260429 20:00:28.788976 1334 flags.cc:432] Enabled experimental flag: --jwks_url=default.url
I20260429 20:00:28.794201 1334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:36267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:36267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:28.795372 1334 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:28.796233 1334 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:28.802659 1339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:28.802891 1340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:28.803126 1342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:28.803113 1334 server_base.cc:1061] running on GCE node
I20260429 20:00:28.803858 1334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:28.804896 1334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:28.806106 1334 hybrid_clock.cc:648] HybridClock initialized: now 1777492828806097 us; error 32 us; skew 500 ppm
I20260429 20:00:28.808178 1334 webserver.cc:492] Webserver started at http://127.27.17.254:40221/ using document root <none> and password file <none>
I20260429 20:00:28.808792 1334 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:28.808878 1334 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:28.809106 1334 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:28.811182 1334 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "f3fefce90d2c450ca5b0df97108c7518"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "c10256d305019e02bca810e6cbf3d131"
server_key_iv: "809e85946a546b19f223f6bff643664f"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.811761 1334 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "f3fefce90d2c450ca5b0df97108c7518"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "c10256d305019e02bca810e6cbf3d131"
server_key_iv: "809e85946a546b19f223f6bff643664f"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.815366 1334 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:28.818006 1348 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:28.819209 1334 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:28.819396 1334 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "f3fefce90d2c450ca5b0df97108c7518"
format_stamp: "Formatted at 2026-04-29 20:00:28 on dist-test-slave-01nr"
server_key: "c10256d305019e02bca810e6cbf3d131"
server_key_iv: "809e85946a546b19f223f6bff643664f"
server_key_version: "encryptionkey@0"
I20260429 20:00:28.819523 1334 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:28.834568 1334 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:28.835474 1334 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:28.835703 1334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:28.843881 1400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:36267 every 8 connection(s)
I20260429 20:00:28.843902 1334 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:36267
I20260429 20:00:28.845031 1334 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:28.848210 1401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:28.849367 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1334
I20260429 20:00:28.849573 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:28.849933 27719 external_mini_cluster.cc:1468] Setting key eb287cf92f2bb42896823acce1d9fb1b
I20260429 20:00:28.855258 1401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518: Bootstrap starting.
I20260429 20:00:28.858294 1401 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:28.859220 1404 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:28.851571 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.27.17.254:36267)
0429 20:00:28.852304 (+ 733us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:28.852314 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:28.852354 (+ 40us) server_negotiation.cc:408] Connection header received
0429 20:00:28.852930 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:28.852947 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:28.853321 (+ 374us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:28.853682 (+ 361us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:28.854252 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.855282 (+ 1030us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.856271 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.856540 (+ 269us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.857254 (+ 714us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:28.857299 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:28.857312 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:00:28.857697 (+ 385us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:28.857733 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:00:28.857746 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:28.857827 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:28.858093 (+ 266us) server_negotiation.cc:300] Negotiation successful
0429 20:00:28.858389 (+ 296us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":154,"threads_started":1}
I20260429 20:00:28.859251 1401 log.cc:826] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:28.861557 1401 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518: No bootstrap required, opened a new log
I20260429 20:00:28.862579 27719 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20260429 20:00:28.865131 1401 raft_consensus.cc:359] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fefce90d2c450ca5b0df97108c7518" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36267 } }
I20260429 20:00:28.865675 1401 raft_consensus.cc:385] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:28.865743 1401 raft_consensus.cc:740] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fefce90d2c450ca5b0df97108c7518, State: Initialized, Role: FOLLOWER
I20260429 20:00:28.866187 1401 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fefce90d2c450ca5b0df97108c7518" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36267 } }
I20260429 20:00:28.866334 1401 raft_consensus.cc:399] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:28.866376 1401 raft_consensus.cc:493] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:28.866448 1401 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:28.867451 1401 raft_consensus.cc:515] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fefce90d2c450ca5b0df97108c7518" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36267 } }
I20260429 20:00:28.867894 1401 leader_election.cc:304] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3fefce90d2c450ca5b0df97108c7518; no voters:
I20260429 20:00:28.868196 1401 leader_election.cc:290] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:28.868444 1406 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:28.868697 1406 raft_consensus.cc:697] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [term 1 LEADER]: Becoming Leader. State: Replica: f3fefce90d2c450ca5b0df97108c7518, State: Running, Role: LEADER
I20260429 20:00:28.869024 1406 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fefce90d2c450ca5b0df97108c7518" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36267 } }
I20260429 20:00:28.869479 1401 sys_catalog.cc:565] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:28.871054 1407 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3fefce90d2c450ca5b0df97108c7518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fefce90d2c450ca5b0df97108c7518" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36267 } } }
I20260429 20:00:28.871186 1407 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:28.871272 1408 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3fefce90d2c450ca5b0df97108c7518. Latest consensus state: current_term: 1 leader_uuid: "f3fefce90d2c450ca5b0df97108c7518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fefce90d2c450ca5b0df97108c7518" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36267 } } }
I20260429 20:00:28.871676 1408 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518 [sys.catalog]: This master's current role is: LEADER
W20260429 20:00:28.875444 1421 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:00:28.875547 1421 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:00:28.875653 1416 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:28.876521 1416 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:28.882577 1416 catalog_manager.cc:1357] Generated new cluster ID: fa085ef66d6441ce89727891de9be923
I20260429 20:00:28.882671 1416 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:28.913398 1416 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:28.914301 1416 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:28.921021 1416 catalog_manager.cc:6044] T 00000000000000000000000000000000 P f3fefce90d2c450ca5b0df97108c7518: Generated new TSK 0
I20260429 20:00:28.921720 1416 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:00:29.016098 1404 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:28.933387 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.27.17.254:36267)
0429 20:00:28.933567 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:28.933572 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:28.933587 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:28.933674 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:28.933677 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:28.933731 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:28.933843 (+ 112us) server_negotiation.cc:227] Negotiated authn=JWT
0429 20:00:28.934233 (+ 390us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.934798 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.935526 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.935733 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.935820 (+ 87us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
0429 20:00:29.015625 (+ 79805us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0429 20:00:29.015941 (+ 316us) negotiation.cc:326] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41188: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":63}
W20260429 20:00:29.016191 1404 negotiation.cc:343] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41188: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20260429 20:00:29.016209 1430 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:28.933044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36267 (local address 127.0.0.1:41188)
0429 20:00:28.933352 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:28.933366 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:28.933490 (+ 124us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:28.933850 (+ 360us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:28.933853 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:28.933864 (+ 11us) client_negotiation.cc:190] Negotiated authn=JWT
0429 20:00:28.934105 (+ 241us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:28.934113 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.934964 (+ 851us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:28.934968 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:28.935397 (+ 429us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:28.935405 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:28.935546 (+ 141us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:28.935583 (+ 37us) client_negotiation.cc:253] Sending JWT_EXCHANGE NegotiatePB request
0429 20:00:29.015855 (+ 80272us) client_negotiation.cc:284] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
0429 20:00:29.016026 (+ 171us) negotiation.cc:326] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.17.254:36267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":139,"threads_started":1}
I20260429 20:00:29.016654 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1334
2026-04-29T20:00:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (371 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (122 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:29 dist-test-slave-01nr krb5kdc[1442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:29 dist-test-slave-01nr krb5kdc[1442](info): set up 2 sockets
Apr 29 20:00:29 dist-test-slave-01nr krb5kdc[1442](info): commencing operation
krb5kdc: starting...
W20260429 20:00:31.324496 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:31 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:31.484500 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34675
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:34675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:31.595114 1458 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:31.595417 1458 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:31.595515 1458 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:31.599177 1458 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:31.599298 1458 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:31.599344 1458 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:31.599408 1458 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:31.599467 1458 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:31.604624 1458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:34675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:31.605825 1458 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:31.606710 1458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:31.612402 1466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:31.612439 1463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:31.612390 1464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:31.612815 1458 server_base.cc:1061] running on GCE node
I20260429 20:00:31.613348 1458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:31.614324 1458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:31.615556 1458 hybrid_clock.cc:648] HybridClock initialized: now 1777492831615526 us; error 53 us; skew 500 ppm
Apr 29 20:00:31 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:31.618361 1458 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:31.619475 1458 webserver.cc:492] Webserver started at http://127.27.17.254:41481/ using document root <none> and password file <none>
I20260429 20:00:31.620066 1458 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:31.620118 1458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:31.620288 1458 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:31.621960 1458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "49c6024b1adc4c9f88f10617cf74ad31"
format_stamp: "Formatted at 2026-04-29 20:00:31 on dist-test-slave-01nr"
server_key: "fafaca55f917224a753144208f5c84a1"
server_key_iv: "95de99ccfd0d49ca961fa186483c8706"
server_key_version: "encryptionkey@0"
I20260429 20:00:31.622416 1458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "49c6024b1adc4c9f88f10617cf74ad31"
format_stamp: "Formatted at 2026-04-29 20:00:31 on dist-test-slave-01nr"
server_key: "fafaca55f917224a753144208f5c84a1"
server_key_iv: "95de99ccfd0d49ca961fa186483c8706"
server_key_version: "encryptionkey@0"
I20260429 20:00:31.625734 1458 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 20:00:31.627893 1473 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:31.628924 1458 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:31.629030 1458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "49c6024b1adc4c9f88f10617cf74ad31"
format_stamp: "Formatted at 2026-04-29 20:00:31 on dist-test-slave-01nr"
server_key: "fafaca55f917224a753144208f5c84a1"
server_key_iv: "95de99ccfd0d49ca961fa186483c8706"
server_key_version: "encryptionkey@0"
I20260429 20:00:31.629114 1458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:31.638604 1458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:31.648150 1458 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:31.648332 1458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:31.656392 1458 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:34675
I20260429 20:00:31.656483 1525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:34675 every 8 connection(s)
I20260429 20:00:31.657754 1458 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:31.660902 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1458
I20260429 20:00:31.661015 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:31.661067 1526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:31.661386 27719 external_mini_cluster.cc:1468] Setting key d0d0e07fd33d08605f1b6e0aa576ae8b
I20260429 20:00:31.670205 1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31: Bootstrap starting.
Apr 29 20:00:31 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:31.673871 1526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:31.674901 1526 log.cc:826] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:31.677615 1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31: No bootstrap required, opened a new log
I20260429 20:00:31.679087 1529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:31.663028 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.27.17.254:34675)
0429 20:00:31.663682 (+ 654us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:31.663695 (+ 13us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:31.663741 (+ 46us) server_negotiation.cc:408] Connection header received
0429 20:00:31.665386 (+ 1645us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:31.665419 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:31.665977 (+ 558us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:31.666449 (+ 472us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:31.667589 (+ 1140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:31.668750 (+ 1161us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:31.669846 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:31.670156 (+ 310us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:31.673554 (+ 3398us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:31.673574 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:31.673593 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:31.673635 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:31.675829 (+ 2194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:31.676442 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:31.676450 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:31.676455 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:31.676575 (+ 120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:31.676886 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:31.676890 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:31.676892 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:31.677337 (+ 445us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:31.677535 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:31.677929 (+ 394us) server_negotiation.cc:300] Negotiation successful
0429 20:00:31.678200 (+ 271us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":118,"threads_started":1}
I20260429 20:00:31.680460 1526 raft_consensus.cc:359] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c6024b1adc4c9f88f10617cf74ad31" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34675 } }
I20260429 20:00:31.680696 1526 raft_consensus.cc:385] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:31.680787 1526 raft_consensus.cc:740] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c6024b1adc4c9f88f10617cf74ad31, State: Initialized, Role: FOLLOWER
I20260429 20:00:31.681268 1526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c6024b1adc4c9f88f10617cf74ad31" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34675 } }
I20260429 20:00:31.681432 1526 raft_consensus.cc:399] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:31.681497 1526 raft_consensus.cc:493] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:31.681589 1526 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:31.683037 1526 raft_consensus.cc:515] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c6024b1adc4c9f88f10617cf74ad31" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34675 } }
I20260429 20:00:31.683429 1526 leader_election.cc:304] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c6024b1adc4c9f88f10617cf74ad31; no voters:
I20260429 20:00:31.683763 1526 leader_election.cc:290] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:31.684036 1531 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:31.684332 1531 raft_consensus.cc:697] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [term 1 LEADER]: Becoming Leader. State: Replica: 49c6024b1adc4c9f88f10617cf74ad31, State: Running, Role: LEADER
I20260429 20:00:31.684661 1531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c6024b1adc4c9f88f10617cf74ad31" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34675 } }
I20260429 20:00:31.685062 1526 sys_catalog.cc:565] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:31.686595 1532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c6024b1adc4c9f88f10617cf74ad31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c6024b1adc4c9f88f10617cf74ad31" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34675 } } }
I20260429 20:00:31.686609 1533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c6024b1adc4c9f88f10617cf74ad31. Latest consensus state: current_term: 1 leader_uuid: "49c6024b1adc4c9f88f10617cf74ad31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c6024b1adc4c9f88f10617cf74ad31" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 34675 } } }
I20260429 20:00:31.686993 1532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:31.686992 1533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:31.687793 1540 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:31.690951 1540 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:31.698601 1540 catalog_manager.cc:1357] Generated new cluster ID: 56e1fdcbef924abaa0acca135cb208e5
I20260429 20:00:31.698689 1540 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:31.715804 1540 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:31.716689 1540 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:31.729148 1540 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 49c6024b1adc4c9f88f10617cf74ad31: Generated new TSK 0
I20260429 20:00:31.729965 1540 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:31.805379 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34675
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:31.921955 1554 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:31.922179 1554 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:31.922240 1554 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:31.925810 1554 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:31.925897 1554 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:31.926017 1554 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:31.930505 1554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:31.931831 1554 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:31.932704 1554 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:31.939723 1560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:31.939723 1559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:31.939723 1562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:31.939998 1554 server_base.cc:1061] running on GCE node
I20260429 20:00:31.940470 1554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:31.941099 1554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:31.942305 1554 hybrid_clock.cc:648] HybridClock initialized: now 1777492831942289 us; error 32 us; skew 500 ppm
Apr 29 20:00:31 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:31.945411 1554 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:00:31.946552 1554 webserver.cc:492] Webserver started at http://127.27.17.193:33961/ using document root <none> and password file <none>
I20260429 20:00:31.947187 1554 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:31.947244 1554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:31.947467 1554 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:31.949239 1554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "b70ecc1492214a2b86d6d19a81a721b0"
format_stamp: "Formatted at 2026-04-29 20:00:31 on dist-test-slave-01nr"
server_key: "c9c0e4c1fd80cdce09b7b68ad35a42b2"
server_key_iv: "ff2fae5eb78c99d9ca7edca6613ea5e6"
server_key_version: "encryptionkey@0"
I20260429 20:00:31.949738 1554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "b70ecc1492214a2b86d6d19a81a721b0"
format_stamp: "Formatted at 2026-04-29 20:00:31 on dist-test-slave-01nr"
server_key: "c9c0e4c1fd80cdce09b7b68ad35a42b2"
server_key_iv: "ff2fae5eb78c99d9ca7edca6613ea5e6"
server_key_version: "encryptionkey@0"
I20260429 20:00:31.953282 1554 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 20:00:31.955626 1569 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:31.956712 1554 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:31.956871 1554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "b70ecc1492214a2b86d6d19a81a721b0"
format_stamp: "Formatted at 2026-04-29 20:00:31 on dist-test-slave-01nr"
server_key: "c9c0e4c1fd80cdce09b7b68ad35a42b2"
server_key_iv: "ff2fae5eb78c99d9ca7edca6613ea5e6"
server_key_version: "encryptionkey@0"
I20260429 20:00:31.956985 1554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:31.994093 1554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:31.997354 1554 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:31.997685 1554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:31.998314 1554 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:31.999346 1554 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:31.999419 1554 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:31.999500 1554 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:31.999544 1554 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.011165 1554 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:33309
I20260429 20:00:32.011229 1682 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:33309 every 8 connection(s)
I20260429 20:00:32.012408 1554 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:32.012732 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1554
I20260429 20:00:32.012872 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:32.013181 27719 external_mini_cluster.cc:1468] Setting key e3eaceebd7aae7e4239d9ca0f9706898
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
I20260429 20:00:32.026443 1529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.014281 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:50425 (local address 127.27.17.254:34675)
0429 20:00:32.014443 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.014447 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.015468 (+ 1021us) server_negotiation.cc:408] Connection header received
0429 20:00:32.016359 (+ 891us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.016367 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.016436 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.016573 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.018432 (+ 1859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.019020 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.019688 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.019875 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.022374 (+ 2499us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:32.022401 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:32.022403 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:32.022462 (+ 59us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:32.024187 (+ 1725us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.024792 (+ 605us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.024801 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.024803 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.024864 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.025224 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.025228 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.025230 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.025397 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:32.025626 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:32.026134 (+ 508us) server_negotiation.cc:300] Negotiation successful
0429 20:00:32.026271 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 20:00:32.027168 1685 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.014797 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34675 (local address 127.27.17.193:50425)
0429 20:00:32.015314 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:32.015355 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:32.016157 (+ 802us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:32.016754 (+ 597us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:32.016766 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:32.017364 (+ 598us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:32.018258 (+ 894us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:32.018276 (+ 18us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.019153 (+ 877us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.019157 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:32.019562 (+ 405us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:32.019568 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.019796 (+ 228us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.020442 (+ 646us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:32.020462 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:32.022200 (+ 1738us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:32.024338 (+ 2138us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:32.024344 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:32.024358 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:32.024614 (+ 256us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:32.025006 (+ 392us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:32.025010 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:32.025014 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:32.025127 (+ 113us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:32.025609 (+ 482us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:32.025616 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:32.025879 (+ 263us) client_negotiation.cc:770] Sending connection context
0429 20:00:32.026099 (+ 220us) client_negotiation.cc:241] Negotiation successful
0429 20:00:32.026350 (+ 251us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":128,"threads_started":1}
I20260429 20:00:32.028527 1683 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34675
I20260429 20:00:32.028820 1683 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:32.029374 1683 heartbeater.cc:507] Master 127.27.17.254:34675 requested a full tablet report, sending...
I20260429 20:00:32.031104 1490 ts_manager.cc:194] Registered new tserver with Master: b70ecc1492214a2b86d6d19a81a721b0 (127.27.17.193:33309)
I20260429 20:00:32.032307 1490 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:50425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:32.075915 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34675
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:32.185916 1690 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:32.186213 1690 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:32.186312 1690 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:32.190086 1690 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:32.190219 1690 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:32.190418 1690 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:32.195425 1690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:32.196653 1690 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:32.197551 1690 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:32.204866 1695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:32.204948 1690 server_base.cc:1061] running on GCE node
W20260429 20:00:32.204857 1696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:32.204857 1698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:32.205648 1690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:32.206295 1690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:32.207471 1690 hybrid_clock.cc:648] HybridClock initialized: now 1777492832207449 us; error 41 us; skew 500 ppm
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.210433 1690 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:00:32.211659 1690 webserver.cc:492] Webserver started at http://127.27.17.194:45823/ using document root <none> and password file <none>
I20260429 20:00:32.212232 1690 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:32.212307 1690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:32.212523 1690 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:32.214278 1690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "b10b39c2d817481489ad9e26f1866efa"
format_stamp: "Formatted at 2026-04-29 20:00:32 on dist-test-slave-01nr"
server_key: "1b218d3581ac1ff3fe3a3162be6affbe"
server_key_iv: "eee611326151df38fbdf28a56bbb4963"
server_key_version: "encryptionkey@0"
I20260429 20:00:32.214789 1690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "b10b39c2d817481489ad9e26f1866efa"
format_stamp: "Formatted at 2026-04-29 20:00:32 on dist-test-slave-01nr"
server_key: "1b218d3581ac1ff3fe3a3162be6affbe"
server_key_iv: "eee611326151df38fbdf28a56bbb4963"
server_key_version: "encryptionkey@0"
I20260429 20:00:32.218248 1690 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:32.220669 1705 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.221856 1690 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260429 20:00:32.222016 1690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "b10b39c2d817481489ad9e26f1866efa"
format_stamp: "Formatted at 2026-04-29 20:00:32 on dist-test-slave-01nr"
server_key: "1b218d3581ac1ff3fe3a3162be6affbe"
server_key_iv: "eee611326151df38fbdf28a56bbb4963"
server_key_version: "encryptionkey@0"
I20260429 20:00:32.222137 1690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:32.256434 1690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:32.259511 1690 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:32.259733 1690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:32.260320 1690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:32.261226 1690 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:32.261301 1690 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.261406 1690 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:32.261456 1690 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.274391 1690 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:46791
I20260429 20:00:32.274437 1818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:46791 every 8 connection(s)
I20260429 20:00:32.275430 1690 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:32.283190 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1690
I20260429 20:00:32.283298 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:32.283579 27719 external_mini_cluster.cc:1468] Setting key 310ba71fab8635d9d4101b489440d594
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:32.288106 1529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.277301 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:50957 (local address 127.27.17.254:34675)
0429 20:00:32.277453 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.277457 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.278007 (+ 550us) server_negotiation.cc:408] Connection header received
0429 20:00:32.278797 (+ 790us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.278800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.278851 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.279003 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.280506 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.281023 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.281751 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.281889 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.284455 (+ 2566us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:32.284482 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:32.284484 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:32.284513 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:32.285983 (+ 1470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.286567 (+ 584us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.286574 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.286576 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.286658 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.287028 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.287031 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.287033 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.287192 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:32.287296 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:32.287803 (+ 507us) server_negotiation.cc:300] Negotiation successful
0429 20:00:32.287940 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 20:00:32.288786 1821 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.277472 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34675 (local address 127.27.17.194:50957)
0429 20:00:32.277862 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:32.277893 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:32.278620 (+ 727us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:32.279122 (+ 502us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:32.279130 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:32.279523 (+ 393us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:32.280349 (+ 826us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:32.280362 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.281171 (+ 809us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.281174 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:32.281638 (+ 464us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:32.281646 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.281821 (+ 175us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.282389 (+ 568us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:32.282410 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:32.284296 (+ 1886us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:32.286142 (+ 1846us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:32.286148 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:32.286162 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:32.286460 (+ 298us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:32.286768 (+ 308us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:32.286772 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:32.286776 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:32.286878 (+ 102us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:32.287306 (+ 428us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:32.287313 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:32.287549 (+ 236us) client_negotiation.cc:770] Sending connection context
0429 20:00:32.287770 (+ 221us) client_negotiation.cc:241] Negotiation successful
0429 20:00:32.288009 (+ 239us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":87,"threads_started":1}
I20260429 20:00:32.289906 1819 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34675
I20260429 20:00:32.290228 1819 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:32.290985 1819 heartbeater.cc:507] Master 127.27.17.254:34675 requested a full tablet report, sending...
I20260429 20:00:32.292224 1490 ts_manager.cc:194] Registered new tserver with Master: b10b39c2d817481489ad9e26f1866efa (127.27.17.194:46791)
I20260429 20:00:32.292843 1490 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:50957
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:32.343454 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:34675
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:32.451817 1826 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:32.452088 1826 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:32.452181 1826 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:32.455628 1826 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:32.455734 1826 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:32.455849 1826 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:32.460181 1826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:34675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:32.461344 1826 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:32.462214 1826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:32.468641 1831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:32.468704 1834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:32.468825 1832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:32.469024 1826 server_base.cc:1061] running on GCE node
I20260429 20:00:32.469425 1826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:32.470073 1826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:32.471284 1826 hybrid_clock.cc:648] HybridClock initialized: now 1777492832471257 us; error 46 us; skew 500 ppm
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.474097 1826 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 20:00:32.475337 1826 webserver.cc:492] Webserver started at http://127.27.17.195:40823/ using document root <none> and password file <none>
I20260429 20:00:32.475852 1826 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:32.475899 1826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:32.476066 1826 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:32.477782 1826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "a7cbe4c69cc243eda31b739839676eec"
format_stamp: "Formatted at 2026-04-29 20:00:32 on dist-test-slave-01nr"
server_key: "08f17c87063333c45f5b0e1a33ac030c"
server_key_iv: "e7ecd4af4c0d9c62c82a08c58f356516"
server_key_version: "encryptionkey@0"
I20260429 20:00:32.478204 1826 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "a7cbe4c69cc243eda31b739839676eec"
format_stamp: "Formatted at 2026-04-29 20:00:32 on dist-test-slave-01nr"
server_key: "08f17c87063333c45f5b0e1a33ac030c"
server_key_iv: "e7ecd4af4c0d9c62c82a08c58f356516"
server_key_version: "encryptionkey@0"
I20260429 20:00:32.481658 1826 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260429 20:00:32.483959 1841 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.485026 1826 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:32.485131 1826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "a7cbe4c69cc243eda31b739839676eec"
format_stamp: "Formatted at 2026-04-29 20:00:32 on dist-test-slave-01nr"
server_key: "08f17c87063333c45f5b0e1a33ac030c"
server_key_iv: "e7ecd4af4c0d9c62c82a08c58f356516"
server_key_version: "encryptionkey@0"
I20260429 20:00:32.485281 1826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:32.512027 1826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:32.515646 1826 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:32.515879 1826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:32.516507 1826 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:32.517427 1826 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:32.517480 1826 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.517525 1826 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:32.517540 1826 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:32.527441 1826 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:37113
I20260429 20:00:32.527457 1954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:37113 every 8 connection(s)
I20260429 20:00:32.528587 1826 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:32.530062 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1826
I20260429 20:00:32.530193 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:32.530503 27719 external_mini_cluster.cc:1468] Setting key 22db56ad2c1919ee7571243019862926
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:32.542133 1529 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.530630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:34063 (local address 127.27.17.254:34675)
0429 20:00:32.530781 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.530786 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.531605 (+ 819us) server_negotiation.cc:408] Connection header received
0429 20:00:32.532772 (+ 1167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.532777 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.532861 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.532951 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.534711 (+ 1760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.535299 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.535954 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.536133 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.538682 (+ 2549us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:32.538710 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:32.538713 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:32.538742 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:32.540253 (+ 1511us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.540819 (+ 566us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.540822 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.540823 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.540866 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.541130 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.541132 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.541133 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.541276 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:32.541370 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:32.541904 (+ 534us) server_negotiation.cc:300] Negotiation successful
0429 20:00:32.542007 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 20:00:32.542793 1957 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.530938 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:34675 (local address 127.27.17.195:34063)
0429 20:00:32.531443 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:32.531510 (+ 67us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:32.532458 (+ 948us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:32.533168 (+ 710us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:32.533178 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:32.533691 (+ 513us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:32.534562 (+ 871us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:32.534574 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.535422 (+ 848us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.535428 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:32.535832 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:32.535838 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.536061 (+ 223us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.536848 (+ 787us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:32.536868 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:32.538470 (+ 1602us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:32.540418 (+ 1948us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:32.540424 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:32.540436 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:32.540724 (+ 288us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:32.540945 (+ 221us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:32.540947 (+ 2us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:32.540949 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:32.541054 (+ 105us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:32.541403 (+ 349us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:32.541410 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:32.541645 (+ 235us) client_negotiation.cc:770] Sending connection context
0429 20:00:32.541845 (+ 200us) client_negotiation.cc:241] Negotiation successful
0429 20:00:32.542045 (+ 200us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260429 20:00:32.544005 1955 heartbeater.cc:344] Connected to a master server at 127.27.17.254:34675
I20260429 20:00:32.544312 1955 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:32.544832 1955 heartbeater.cc:507] Master 127.27.17.254:34675 requested a full tablet report, sending...
I20260429 20:00:32.545827 1490 ts_manager.cc:194] Registered new tserver with Master: a7cbe4c69cc243eda31b739839676eec (127.27.17.195:37113)
I20260429 20:00:32.546382 1490 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:34063
I20260429 20:00:32.556699 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/test-admin.keytab.
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.576152 27719 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260429 20:00:32.578979 1962 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:32.577562 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.27.17.193:33309)
0429 20:00:32.577858 (+ 296us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.577864 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.577886 (+ 22us) server_negotiation.cc:408] Connection header received
0429 20:00:32.577938 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.577942 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.578093 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.578205 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.578747 (+ 542us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36340: BlockingRecv error: recv got EOF from 127.0.0.1:36340 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":96,"threads_started":1}
W20260429 20:00:32.579160 27719 init.cc:244] Error while opening credential cache '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc)
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.580236 27719 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.581202 27719 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 20:00:32.591089 1962 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.582253 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36346 (local address 127.27.17.193:33309)
0429 20:00:32.582408 (+ 155us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.582415 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.582433 (+ 18us) server_negotiation.cc:408] Connection header received
0429 20:00:32.582485 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.582488 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.582538 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.582628 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.583396 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.584153 (+ 757us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.584839 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.585101 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.587357 (+ 2256us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:32.587392 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:32.587404 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:32.587432 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:32.589336 (+ 1904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.589818 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.589823 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.589825 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.589877 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.590128 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.590132 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.590134 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.590326 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:32.590538 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:32.590760 (+ 222us) server_negotiation.cc:300] Negotiation successful
0429 20:00:32.590917 (+ 157us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260429 20:00:32.595149 1962 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:32.594083 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36362 (local address 127.27.17.193:33309)
0429 20:00:32.594205 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.594209 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.594249 (+ 40us) server_negotiation.cc:408] Connection header received
0429 20:00:32.594434 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.594437 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.594484 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.594563 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.594998 (+ 435us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36362: BlockingRecv error: recv got EOF from 127.0.0.1:36362 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260429 20:00:32.595235 27719 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc' of type FILE
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.596310 27719 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.597208 27719 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 20:00:32.606448 1962 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.598113 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.27.17.193:33309)
0429 20:00:32.598243 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.598246 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.598261 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:32.598452 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.598455 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.598510 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.598589 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.599364 (+ 775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.599900 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.600686 (+ 786us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.600908 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.603490 (+ 2582us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:32.603514 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:32.603516 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:32.603548 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:32.605037 (+ 1489us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.605449 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.605453 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.605454 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.605506 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.605771 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.605774 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.605775 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.605908 (+ 133us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:32.606006 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:32.606195 (+ 189us) server_negotiation.cc:300] Negotiation successful
0429 20:00:32.606319 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260429 20:00:32.609426 1962 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:32.608440 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.27.17.193:33309)
0429 20:00:32.608610 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.608613 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.608627 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:32.608720 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.608723 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.608764 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.608833 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.609316 (+ 483us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36384: BlockingRecv error: recv got EOF from 127.0.0.1:36384 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260429 20:00:32.609557 27719 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1777492770945061-27719-0/krb5kdc/krb5cc' of type FILE
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.610677 27719 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:32.611723 27719 init.cc:295] Successfully reacquired a new kerberos TGT
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492832, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.193@KRBTEST.COM
I20260429 20:00:32.620398 1962 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:32.612648 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.27.17.193:33309)
0429 20:00:32.612779 (+ 131us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:32.612783 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:32.612796 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:32.612988 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:32.612991 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:32.613032 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:32.613114 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:32.613766 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.614230 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:32.614859 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:32.615195 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:32.617165 (+ 1970us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:32.617184 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:32.617187 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:32.617214 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:32.618668 (+ 1454us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.619191 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.619194 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.619195 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.619240 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:32.619532 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:32.619536 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:32.619537 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:32.619680 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:32.619803 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:32.620059 (+ 256us) server_negotiation.cc:300] Negotiation successful
0429 20:00:32.620161 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 20:00:32.621776 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1554
I20260429 20:00:32.627614 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1690
I20260429 20:00:32.633556 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1826
I20260429 20:00:32.639508 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1458
2026-04-29T20:00:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3376 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1975](info): set up 2 sockets
Apr 29 20:00:32 dist-test-slave-01nr krb5kdc[1975](info): commencing operation
krb5kdc: starting...
W20260429 20:00:34.700217 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:34 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:34.863921 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45045
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:45045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:34.974352 1991 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:34.974642 1991 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:34.974730 1991 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:34.978193 1991 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:34.978320 1991 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:34.978367 1991 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:34.978430 1991 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:34.978471 1991 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:34.983050 1991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:45045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:34.984310 1991 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:34.985452 1991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:34.991679 1996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:34.991679 1999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:34.991694 1997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:34.992090 1991 server_base.cc:1061] running on GCE node
I20260429 20:00:34.992650 1991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:34.993660 1991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:34.994834 1991 hybrid_clock.cc:648] HybridClock initialized: now 1777492834994810 us; error 45 us; skew 500 ppm
Apr 29 20:00:34 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:34.997957 1991 init.cc:377] Logged in from keytab as oryx/127.27.17.254@KRBTEST.COM (short username oryx)
I20260429 20:00:34.999167 1991 webserver.cc:492] Webserver started at http://127.27.17.254:35015/ using document root <none> and password file <none>
I20260429 20:00:34.999765 1991 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:34.999821 1991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:35.000038 1991 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:35.001809 1991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "4a284ad344d34d8c9e4cf992f4bb42d4"
format_stamp: "Formatted at 2026-04-29 20:00:34 on dist-test-slave-01nr"
server_key: "92b3159ed01c5c4fd84f7e99d6254af3"
server_key_iv: "428a7fca489f0a8bbae1d9a2d581be9a"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.002364 1991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "4a284ad344d34d8c9e4cf992f4bb42d4"
format_stamp: "Formatted at 2026-04-29 20:00:34 on dist-test-slave-01nr"
server_key: "92b3159ed01c5c4fd84f7e99d6254af3"
server_key_iv: "428a7fca489f0a8bbae1d9a2d581be9a"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.006119 1991 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 20:00:35.008992 2006 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.010340 1991 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260429 20:00:35.010593 1991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "4a284ad344d34d8c9e4cf992f4bb42d4"
format_stamp: "Formatted at 2026-04-29 20:00:34 on dist-test-slave-01nr"
server_key: "92b3159ed01c5c4fd84f7e99d6254af3"
server_key_iv: "428a7fca489f0a8bbae1d9a2d581be9a"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.010747 1991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:35.034632 1991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:35.038038 1991 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:35.038283 1991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:35.046205 1991 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:45045
I20260429 20:00:35.046207 2058 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:45045 every 8 connection(s)
I20260429 20:00:35.047506 1991 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:35.050377 2059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:35.050848 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 1991
I20260429 20:00:35.051021 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:35.051287 27719 external_mini_cluster.cc:1468] Setting key b8993fb4fa367665f26554b3fc0f60d9
I20260429 20:00:35.057888 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Bootstrap starting.
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:35.060674 2059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:35.061560 2059 log.cc:826] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:35.063555 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: No bootstrap required, opened a new log
I20260429 20:00:35.066265 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.052725 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.27.17.254:45045)
0429 20:00:35.053200 (+ 475us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:35.053215 (+ 15us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:35.053261 (+ 46us) server_negotiation.cc:408] Connection header received
0429 20:00:35.054214 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:35.054241 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:35.054640 (+ 399us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:35.055038 (+ 398us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:35.055969 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.057153 (+ 1184us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.058130 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.058434 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.060986 (+ 2552us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:35.061004 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:35.061017 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:35.061050 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:35.063184 (+ 2134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.063829 (+ 645us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.063835 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.063840 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.063932 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.064209 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.064213 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.064214 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.064622 (+ 408us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:35.064805 (+ 183us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:35.065132 (+ 327us) server_negotiation.cc:300] Negotiation successful
0429 20:00:35.065370 (+ 238us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20260429 20:00:35.067103 2059 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } }
I20260429 20:00:35.067307 2059 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:35.067415 2059 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a284ad344d34d8c9e4cf992f4bb42d4, State: Initialized, Role: FOLLOWER
I20260429 20:00:35.067843 2059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } }
I20260429 20:00:35.067963 2059 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:35.068048 2059 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:35.068140 2059 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:35.069646 2059 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } }
I20260429 20:00:35.070027 2059 leader_election.cc:304] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a284ad344d34d8c9e4cf992f4bb42d4; no voters:
I20260429 20:00:35.070354 2059 leader_election.cc:290] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:35.070519 2064 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:35.070766 2064 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 LEADER]: Becoming Leader. State: Replica: 4a284ad344d34d8c9e4cf992f4bb42d4, State: Running, Role: LEADER
I20260429 20:00:35.071161 2064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } }
I20260429 20:00:35.071766 2059 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:35.072923 2066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a284ad344d34d8c9e4cf992f4bb42d4. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } }
I20260429 20:00:35.073058 2066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:35.073482 2065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } }
I20260429 20:00:35.073606 2073 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:35.073648 2065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:35.078013 2073 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:35.083649 2073 catalog_manager.cc:1357] Generated new cluster ID: dae3886cdad348fc95dcb8ee81b09b07
I20260429 20:00:35.083768 2073 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:35.093559 2073 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:35.094964 2073 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:35.119144 2073 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Generated new TSK 0
I20260429 20:00:35.119946 2073 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:35.182998 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45045
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:35.294024 2087 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:35.294289 2087 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:35.294350 2087 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:35.298077 2087 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:35.298151 2087 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:35.298270 2087 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:35.302979 2087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:35.304054 2087 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:35.304878 2087 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:35.313005 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:35.312997 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:35.312999 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:35.313426 2087 server_base.cc:1061] running on GCE node
I20260429 20:00:35.313848 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:35.314445 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:35.315623 2087 hybrid_clock.cc:648] HybridClock initialized: now 1777492835315594 us; error 42 us; skew 500 ppm
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492835, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:35.318799 2087 init.cc:377] Logged in from keytab as oryx/127.27.17.193@KRBTEST.COM (short username oryx)
I20260429 20:00:35.320006 2087 webserver.cc:492] Webserver started at http://127.27.17.193:42345/ using document root <none> and password file <none>
I20260429 20:00:35.320632 2087 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:35.320715 2087 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:35.320941 2087 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:35.322695 2087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "dc242720f061411e98271e1a9ad950b1"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "752711b626a44d2727022337fb694896"
server_key_iv: "7625522bdd7c7af1d7eb433f6d4ebc61"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.323233 2087 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "dc242720f061411e98271e1a9ad950b1"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "752711b626a44d2727022337fb694896"
server_key_iv: "7625522bdd7c7af1d7eb433f6d4ebc61"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.326771 2087 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260429 20:00:35.329201 2102 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.330420 2087 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:35.330613 2087 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "dc242720f061411e98271e1a9ad950b1"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "752711b626a44d2727022337fb694896"
server_key_iv: "7625522bdd7c7af1d7eb433f6d4ebc61"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.330740 2087 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:35.382367 2087 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:35.385548 2087 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:35.385763 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:35.386369 2087 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:35.387362 2087 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:35.387434 2087 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.387523 2087 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:35.387575 2087 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.398165 2087 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:34583
I20260429 20:00:35.398228 2215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:34583 every 8 connection(s)
I20260429 20:00:35.399258 2087 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:35.400530 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 2087
I20260429 20:00:35.400683 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:35.400986 27719 external_mini_cluster.cc:1468] Setting key 5f0d3b9c0c8e670d0d28091dd14362bc
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492835, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 20:00:35.416080 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.401630 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:37419 (local address 127.27.17.254:45045)
0429 20:00:35.401829 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:35.401833 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:35.403182 (+ 1349us) server_negotiation.cc:408] Connection header received
0429 20:00:35.404146 (+ 964us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:35.404149 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:35.404198 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:35.404296 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:35.405928 (+ 1632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.406556 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.407675 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.407832 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.411853 (+ 4021us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:35.411886 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:35.411889 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:35.411922 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:35.413527 (+ 1605us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.414252 (+ 725us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.414256 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.414257 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.414312 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.414733 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.414737 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.414738 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.414998 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:35.415118 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:35.415767 (+ 649us) server_negotiation.cc:300] Negotiation successful
0429 20:00:35.415944 (+ 177us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260429 20:00:35.416674 2218 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.402010 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:37419)
0429 20:00:35.402728 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:35.402774 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:35.403883 (+ 1109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:35.404503 (+ 620us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:35.404519 (+ 16us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:35.405136 (+ 617us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:35.405735 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:35.405749 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.406767 (+ 1018us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.406775 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:35.407491 (+ 716us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:35.407506 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.407789 (+ 283us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.409323 (+ 1534us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:35.409358 (+ 35us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:35.411649 (+ 2291us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:35.413684 (+ 2035us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:35.413693 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:35.413709 (+ 16us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:35.413984 (+ 275us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:35.414467 (+ 483us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:35.414470 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:35.414472 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:35.414590 (+ 118us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:35.415186 (+ 596us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:35.415192 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:35.415466 (+ 274us) client_negotiation.cc:770] Sending connection context
0429 20:00:35.415671 (+ 205us) client_negotiation.cc:241] Negotiation successful
0429 20:00:35.415894 (+ 223us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.194@KRBTEST.COM" created.
I20260429 20:00:35.418339 2216 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45045
I20260429 20:00:35.418637 2216 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:35.419354 2216 heartbeater.cc:507] Master 127.27.17.254:45045 requested a full tablet report, sending...
I20260429 20:00:35.421171 2023 ts_manager.cc:194] Registered new tserver with Master: dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583)
I20260429 20:00:35.422465 2023 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:37419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:35.464282 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45045
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:35.569056 2223 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:35.569350 2223 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:35.569445 2223 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:35.573417 2223 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:35.573647 2223 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:35.573804 2223 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:35.578599 2223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:35.579777 2223 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:35.580621 2223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:35.588285 2229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:35.588385 2231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:35.588583 2223 server_base.cc:1061] running on GCE node
W20260429 20:00:35.588590 2228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:35.589123 2223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:35.589848 2223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:35.590996 2223 hybrid_clock.cc:648] HybridClock initialized: now 1777492835590971 us; error 94 us; skew 500 ppm
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492835, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:35.594233 2223 init.cc:377] Logged in from keytab as oryx/127.27.17.194@KRBTEST.COM (short username oryx)
I20260429 20:00:35.595501 2223 webserver.cc:492] Webserver started at http://127.27.17.194:43281/ using document root <none> and password file <none>
I20260429 20:00:35.596118 2223 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:35.596168 2223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:35.596392 2223 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:35.598663 2223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "154fd86362f8596f8ad80fd69d8bb8ca"
server_key_iv: "dab4cb1d1021b56df9f01692ce548dcc"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.599260 2223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "154fd86362f8596f8ad80fd69d8bb8ca"
server_key_iv: "dab4cb1d1021b56df9f01692ce548dcc"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.603045 2223 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 20:00:35.605679 2238 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.606808 2223 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:35.607020 2223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "154fd86362f8596f8ad80fd69d8bb8ca"
server_key_iv: "dab4cb1d1021b56df9f01692ce548dcc"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.607154 2223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:35.619971 2223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:35.623358 2223 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:35.623586 2223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:35.624174 2223 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:35.625106 2223 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:35.625180 2223 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.625252 2223 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:35.625305 2223 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.635671 2223 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:44123
I20260429 20:00:35.635702 2351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:44123 every 8 connection(s)
I20260429 20:00:35.636667 2223 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:35.640741 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 2223
I20260429 20:00:35.640837 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:35.641070 27719 external_mini_cluster.cc:1468] Setting key 3f65f24948d27345a0f225fcb7a192e0
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492835, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 20:00:35.651793 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.638561 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:54899 (local address 127.27.17.254:45045)
0429 20:00:35.638724 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:35.638728 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:35.639442 (+ 714us) server_negotiation.cc:408] Connection header received
0429 20:00:35.640273 (+ 831us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:35.640277 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:35.640330 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:35.640411 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:35.641735 (+ 1324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.642323 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.643067 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.643257 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.647096 (+ 3839us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:35.647127 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:35.647133 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:35.647178 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:35.649047 (+ 1869us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.649816 (+ 769us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.649821 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.649823 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.649884 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.650566 (+ 682us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.650577 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.650579 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.650751 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:35.650874 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:35.651447 (+ 573us) server_negotiation.cc:300] Negotiation successful
0429 20:00:35.651617 (+ 170us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:35.652495 2354 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.638846 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:54899)
0429 20:00:35.639308 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:35.639341 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:35.640047 (+ 706us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:35.640558 (+ 511us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:35.640566 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:35.641008 (+ 442us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:35.641591 (+ 583us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:35.641611 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.642446 (+ 835us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.642450 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:35.642877 (+ 427us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:35.642884 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.643182 (+ 298us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.644665 (+ 1483us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:35.644686 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:35.646807 (+ 2121us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:35.649208 (+ 2401us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:35.649214 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:35.649225 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:35.649665 (+ 440us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:35.650043 (+ 378us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:35.650046 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:35.650048 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:35.650175 (+ 127us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:35.650875 (+ 700us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:35.650884 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:35.651176 (+ 292us) client_negotiation.cc:770] Sending connection context
0429 20:00:35.651407 (+ 231us) client_negotiation.cc:241] Negotiation successful
0429 20:00:35.651674 (+ 267us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":97,"threads_started":1}
I20260429 20:00:35.653934 2352 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45045
I20260429 20:00:35.654340 2352 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.195@KRBTEST.COM" created.
I20260429 20:00:35.655110 2352 heartbeater.cc:507] Master 127.27.17.254:45045 requested a full tablet report, sending...
I20260429 20:00:35.656497 2023 ts_manager.cc:194] Registered new tserver with Master: b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123)
I20260429 20:00:35.657236 2023 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:54899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:35.704353 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45045
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:35.810724 2359 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:35.811007 2359 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:35.811219 2359 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:35.815445 2359 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:35.815539 2359 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:35.815793 2359 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:35.820590 2359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:45045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:35.821772 2359 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:35.822604 2359 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:35.830425 2364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:35.830356 2365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:35.830356 2367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:35.830643 2359 server_base.cc:1061] running on GCE node
I20260429 20:00:35.831223 2359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:35.832026 2359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:35.833210 2359 hybrid_clock.cc:648] HybridClock initialized: now 1777492835833171 us; error 53 us; skew 500 ppm
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492835, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:35.836570 2359 init.cc:377] Logged in from keytab as oryx/127.27.17.195@KRBTEST.COM (short username oryx)
I20260429 20:00:35.837865 2359 webserver.cc:492] Webserver started at http://127.27.17.195:43465/ using document root <none> and password file <none>
I20260429 20:00:35.838505 2359 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:35.838593 2359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:35.838824 2359 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:35.840912 2359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "6e4282a3737d49509f293b9ffaf9d4cc"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "adbe742aea67d6c6a704aaa55586e980"
server_key_iv: "4cd499f55bfd327c3e1bd79ae789021a"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.841451 2359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "6e4282a3737d49509f293b9ffaf9d4cc"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "adbe742aea67d6c6a704aaa55586e980"
server_key_iv: "4cd499f55bfd327c3e1bd79ae789021a"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.845314 2359 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260429 20:00:35.848398 2374 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.849658 2359 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260429 20:00:35.849915 2359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "6e4282a3737d49509f293b9ffaf9d4cc"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "adbe742aea67d6c6a704aaa55586e980"
server_key_iv: "4cd499f55bfd327c3e1bd79ae789021a"
server_key_version: "encryptionkey@0"
I20260429 20:00:35.850044 2359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:35.877236 2359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:35.880702 2359 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:35.880936 2359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:35.881570 2359 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:35.882515 2359 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:35.882591 2359 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.882663 2359 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:35.882714 2359 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:35.896891 2359 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:41057
I20260429 20:00:35.896945 2487 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:41057 every 8 connection(s)
I20260429 20:00:35.898097 2359 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:35.901588 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 2359
I20260429 20:00:35.901859 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:35.902144 27719 external_mini_cluster.cc:1468] Setting key 87945e00c04dfcec8d2e808f7facc3aa
Apr 29 20:00:35 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492835, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:35.913910 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.900218 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:44149 (local address 127.27.17.254:45045)
0429 20:00:35.900389 (+ 171us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:35.900393 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:35.901145 (+ 752us) server_negotiation.cc:408] Connection header received
0429 20:00:35.902287 (+ 1142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:35.902292 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:35.902365 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:35.902470 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:35.903899 (+ 1429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.904728 (+ 829us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.905826 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.906025 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.909205 (+ 3180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:35.909269 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:35.909273 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:35.909306 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:35.911448 (+ 2142us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.912191 (+ 743us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.912195 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.912197 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.912298 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:35.912716 (+ 418us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:35.912720 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:35.912722 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:35.912901 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:35.913020 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:35.913595 (+ 575us) server_negotiation.cc:300] Negotiation successful
0429 20:00:35.913737 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260429 20:00:35.914630 2490 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:35.900525 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:44149)
0429 20:00:35.900977 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:35.901012 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:35.901966 (+ 954us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:35.902649 (+ 683us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:35.902658 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:35.903219 (+ 561us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:35.903745 (+ 526us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:35.903759 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.904932 (+ 1173us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:35.904940 (+ 8us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:35.905438 (+ 498us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:35.905529 (+ 91us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:35.905740 (+ 211us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:35.907026 (+ 1286us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:35.907050 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:35.908965 (+ 1915us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:35.911687 (+ 2722us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:35.911699 (+ 12us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:35.911718 (+ 19us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:35.912040 (+ 322us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:35.912424 (+ 384us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:35.912429 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:35.912436 (+ 7us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:35.912606 (+ 170us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:35.913021 (+ 415us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:35.913031 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:35.913295 (+ 264us) client_negotiation.cc:770] Sending connection context
0429 20:00:35.913554 (+ 259us) client_negotiation.cc:241] Negotiation successful
0429 20:00:35.913808 (+ 254us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":105,"threads_started":1}
I20260429 20:00:35.916002 2488 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45045
I20260429 20:00:35.916327 2488 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:35.916993 2488 heartbeater.cc:507] Master 127.27.17.254:45045 requested a full tablet report, sending...
I20260429 20:00:35.918135 2023 ts_manager.cc:194] Registered new tserver with Master: 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057)
I20260429 20:00:35.918849 2023 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:44149
I20260429 20:00:35.929941 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.17.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:35.992106 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.253:43877
--webserver_interface=127.27.17.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:45045,127.27.17.253:43877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:36.101511 2495 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:36.101800 2495 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:36.101917 2495 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:36.105407 2495 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:36.105508 2495 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:36.105556 2495 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:36.105618 2495 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:36.105677 2495 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:36.110160 2495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:45045,127.27.17.253:43877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.253:43877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.17.253
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
W20260429 20:00:36.110441 2495 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.17.254:45045,127.27.17.253:43877'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260429 20:00:36.111508 2495 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:36.112469 2495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:36.118934 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.118958 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.118956 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:36.119270 2495 server_base.cc:1061] running on GCE node
I20260429 20:00:36.119804 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:36.120981 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:36.122211 2495 hybrid_clock.cc:648] HybridClock initialized: now 1777492836122171 us; error 52 us; skew 500 ppm
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:36.125356 2495 init.cc:377] Logged in from keytab as oryx/127.27.17.253@KRBTEST.COM (short username oryx)
I20260429 20:00:36.126505 2495 webserver.cc:492] Webserver started at http://127.27.17.253:46075/ using document root <none> and password file <none>
I20260429 20:00:36.127337 2495 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:36.127414 2495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:36.127667 2495 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:36.130231 2495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/instance:
uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "48879457bbadb3b2361882d607cd1471"
server_key_iv: "b13297a3423cdac17b0e4882faf2f757"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.130927 2495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal/instance:
uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "48879457bbadb3b2361882d607cd1471"
server_key_iv: "b13297a3423cdac17b0e4882faf2f757"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.136150 2495 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260429 20:00:36.139434 2510 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:36.140939 2495 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:36.141170 2495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "48879457bbadb3b2361882d607cd1471"
server_key_iv: "b13297a3423cdac17b0e4882faf2f757"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.141337 2495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:36.158205 2495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:36.161521 2495 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:36.161731 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:36.172719 2495 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.253:43877
I20260429 20:00:36.172719 2562 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.253:43877 every 8 connection(s)
I20260429 20:00:36.174455 2495 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/info.pb
I20260429 20:00:36.177546 2563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:36.178777 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 2495
I20260429 20:00:36.182324 2563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.253@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:36.195348 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.184549 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37528 (local address 127.27.17.254:45045)
0429 20:00:36.184713 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.184717 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.185065 (+ 348us) server_negotiation.cc:408] Connection header received
0429 20:00:36.185217 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.185223 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.185279 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.185394 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.186422 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.187197 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.188260 (+ 1063us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.188495 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.190829 (+ 2334us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.190853 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.190857 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.190924 (+ 67us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.192652 (+ 1728us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.193372 (+ 720us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.193376 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.193378 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.193443 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.193912 (+ 469us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.193919 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.193921 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.194195 (+ 274us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.194277 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.194994 (+ 717us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.195153 (+ 159us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 20:00:36.196593 2566 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.180380 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.27.17.253:43877)
0429 20:00:36.180954 (+ 574us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.180970 (+ 16us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.181030 (+ 60us) server_negotiation.cc:408] Connection header received
0429 20:00:36.181901 (+ 871us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.181934 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.182356 (+ 422us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.182937 (+ 581us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.184018 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.185289 (+ 1271us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.187128 (+ 1839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.187436 (+ 308us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.190367 (+ 2931us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.190394 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.190407 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.190436 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.193223 (+ 2787us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.193773 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.193782 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.193787 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.193839 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.194153 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.194156 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.194158 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.194690 (+ 532us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.194890 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.195247 (+ 357us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.195672 (+ 425us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"spinlock_wait_cycles":1280,"thread_start_us":177,"threads_started":1}
I20260429 20:00:36.196593 2567 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.184499 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37528)
0429 20:00:36.184960 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.185000 (+ 40us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.185117 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.185456 (+ 339us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.185460 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.185818 (+ 358us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.186146 (+ 328us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.186156 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.187595 (+ 1439us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.187601 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.188094 (+ 493us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.188102 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.188278 (+ 176us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.188799 (+ 521us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.188817 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.190662 (+ 1845us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.192794 (+ 2132us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.192803 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.192814 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.193223 (+ 409us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.193597 (+ 374us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.193600 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.193601 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.193738 (+ 137us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.194391 (+ 653us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.194399 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.194675 (+ 276us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.194884 (+ 209us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.195733 (+ 849us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":288,"thread_start_us":152,"threads_started":1}
I20260429 20:00:36.198570 2563 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.253@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
I20260429 20:00:36.206125 2567 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.199140 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34206)
0429 20:00:36.199270 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.199285 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.199368 (+ 83us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.199574 (+ 206us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.199577 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.199757 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.199931 (+ 174us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.199936 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.200630 (+ 694us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.200636 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.201013 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.201019 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.201146 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.201704 (+ 558us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.201717 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.203072 (+ 1355us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.204789 (+ 1717us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.204795 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.204797 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.204965 (+ 168us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.205198 (+ 233us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.205203 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.205205 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.205251 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.205618 (+ 367us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.205622 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.205695 (+ 73us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.205799 (+ 104us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.205935 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260429 20:00:36.206125 2566 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.199289 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.27.17.253:43877)
0429 20:00:36.199377 (+ 88us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.199380 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.199394 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:36.199434 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.199440 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.199480 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.199562 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.200043 (+ 481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.200525 (+ 482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.201122 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.201327 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.203189 (+ 1862us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.203213 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.203215 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.203242 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.204629 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.205061 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.205066 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.205068 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.205111 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.205344 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.205349 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.205351 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.205519 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.205620 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.205843 (+ 223us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.206008 (+ 165us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":31}
I20260429 20:00:36.209285 2563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Bootstrap starting.
I20260429 20:00:36.211859 2563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:36.212826 2563 log.cc:826] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:36.214740 2563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: No bootstrap required, opened a new log
I20260429 20:00:36.217360 2563 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } }
I20260429 20:00:36.217569 2563 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:36.217651 2563 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee5c1d0c7ac4d118e868a8aa6a57ce3, State: Initialized, Role: FOLLOWER
I20260429 20:00:36.218075 2563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } }
I20260429 20:00:36.218533 2569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } }
I20260429 20:00:36.218654 2569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:36.219260 2563 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: configured and running, proceeding with master startup.
W20260429 20:00:36.224319 2580 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:00:36.224422 2580 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:00:36.226439 2495 master_runner.cc:428] Detected that this master 4ee5c1d0c7ac4d118e868a8aa6a57ce3 is joining an existing cluster
I20260429 20:00:36.226519 2495 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.17.253:43877
I20260429 20:00:36.236424 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.228799 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37534 (local address 127.27.17.254:45045)
0429 20:00:36.229096 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.229101 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.229478 (+ 377us) server_negotiation.cc:408] Connection header received
0429 20:00:36.229607 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.229612 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.229679 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.229790 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.230493 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.231332 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.232012 (+ 680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.232218 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.233048 (+ 830us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.233069 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.233072 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.233117 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.234811 (+ 1694us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.235306 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.235311 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.235313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.235367 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.235688 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.235691 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.235693 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.235840 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.235925 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.236203 (+ 278us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.236297 (+ 94us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260429 20:00:36.236479 2587 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.228658 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37534)
0429 20:00:36.229374 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.229389 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.229500 (+ 111us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.229808 (+ 308us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.229811 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.230118 (+ 307us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.230368 (+ 250us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.230375 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.231479 (+ 1104us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.231483 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.231895 (+ 412us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.231902 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.232003 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.232594 (+ 591us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.232608 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.232909 (+ 301us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.235006 (+ 2097us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.235009 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.235014 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.235202 (+ 188us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.235485 (+ 283us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.235489 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.235501 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.235588 (+ 87us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.235940 (+ 352us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.235943 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.236037 (+ 94us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.236154 (+ 117us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.236287 (+ 133us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":136,"threads_started":1}
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.254@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
I20260429 20:00:36.251295 2589 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.241288 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.27.17.253:43877)
0429 20:00:36.241584 (+ 296us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.241588 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.241725 (+ 137us) server_negotiation.cc:408] Connection header received
0429 20:00:36.241915 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.241918 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.241972 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.242067 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.242831 (+ 764us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.244178 (+ 1347us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.244841 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.245043 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.247206 (+ 2163us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.247226 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.247230 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.247256 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.249410 (+ 2154us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.249991 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.249995 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.249997 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.250044 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.250281 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.250285 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.250287 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.250440 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.250540 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.250973 (+ 433us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.251085 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"spinlock_wait_cycles":12544,"thread_start_us":105,"threads_started":1}
I20260429 20:00:36.251503 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.241245 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34212)
0429 20:00:36.241605 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.241640 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.241783 (+ 143us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.242119 (+ 336us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.242124 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.242495 (+ 371us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.242714 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.242721 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.244368 (+ 1647us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.244372 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.244733 (+ 361us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.244739 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.244836 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.245380 (+ 544us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.245393 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.247069 (+ 1676us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.249561 (+ 2492us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.249571 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.249577 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.249877 (+ 300us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.250134 (+ 257us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.250137 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.250140 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.250193 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.250570 (+ 377us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.250580 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.250711 (+ 131us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.250880 (+ 169us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.251083 (+ 203us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":111,"threads_started":1}
I20260429 20:00:36.253104 2023 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.27.17.253:43877: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
member_type: NON_VOTER
last_known_addr {
host: "127.27.17.253"
port: 43877
}
attrs {
promote: true
}
}
dest_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4"
cas_config_opid_index: -1
I20260429 20:00:36.253888 2023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } }
I20260429 20:00:36.256319 2586 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.229119 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34208)
0429 20:00:36.229409 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.229422 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.229510 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.229808 (+ 298us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.229811 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.247073 (+ 17262us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.247333 (+ 260us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.247339 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.248181 (+ 842us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.248187 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.248603 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.248611 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.248770 (+ 159us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.249320 (+ 550us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.249331 (+ 11us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.249695 (+ 364us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.254032 (+ 4337us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.254037 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.254039 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.254287 (+ 248us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.255082 (+ 795us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.255085 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.255093 (+ 8us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.255143 (+ 50us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.255774 (+ 631us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.255777 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.255857 (+ 80us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.255969 (+ 112us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.256106 (+ 137us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
I20260429 20:00:36.257118 2566 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.229208 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34208 (local address 127.27.17.253:43877)
0429 20:00:36.229328 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.229333 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.229473 (+ 140us) server_negotiation.cc:408] Connection header received
0429 20:00:36.229607 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.229611 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.229646 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.229689 (+ 43us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.247454 (+ 17765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.248005 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.248736 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.249161 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.249815 (+ 654us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.249829 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.249833 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.249855 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.253889 (+ 4034us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.254871 (+ 982us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.254875 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.254877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.254953 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.255233 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.255236 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.255238 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.255639 (+ 401us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.255739 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.256860 (+ 1121us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.256976 (+ 116us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260429 20:00:36.257437 2534 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:36.259315 2534 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Refusing update from remote peer 4a284ad344d34d8c9e4cf992f4bb42d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260429 20:00:36.259850 2592 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.261564 2593 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.27.17.252@KRBTEST.COM; defaulting to no policy
I20260429 20:00:36.262018 2592 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.27.17.253:43877
I20260429 20:00:36.262873 2495 master.cc:562] Master@127.27.17.253:43877 shutting down...
I20260429 20:00:36.263005 2594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.263111 2594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
W20260429 20:00:36.264060 2010 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.252@KRBTEST.COM" created.
W20260429 20:00:36.266137 2010 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 -> Peer 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253:43877): Couldn't send request to peer 4ee5c1d0c7ac4d118e868a8aa6a57ce3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260429 20:00:36.269953 2495 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:36.270288 2495 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:36.270433 2495 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: stopping tablet replica
I20260429 20:00:36.276762 2495 master.cc:584] Master@127.27.17.253:43877 shutdown complete.
I20260429 20:00:36.276834 2495 master_runner.cc:305] Clearing existing system tablet
I20260429 20:00:36.277087 2495 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:36.277154 2495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:36.278450 2495 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260429 20:00:36.279696 2600 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:36.279989 2495 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:36.280104 2495 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:36.280151 2495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "48879457bbadb3b2361882d607cd1471"
server_key_iv: "b13297a3423cdac17b0e4882faf2f757"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.282476 2495 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:36.286334 2495 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260429 20:00:36.286430 2495 log.cc:1199] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260429 20:00:36.286684 2495 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Deleting consensus metadata
I20260429 20:00:36.287710 2495 master_runner.cc:315] Copying system tablet from 127.27.17.254:45045
I20260429 20:00:36.289146 2495 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.254:45045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:36.297453 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.289950 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37548 (local address 127.27.17.254:45045)
0429 20:00:36.290119 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.290123 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.290217 (+ 94us) server_negotiation.cc:408] Connection header received
0429 20:00:36.290414 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.290417 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.290470 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.290555 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.291460 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.292241 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.293127 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.293746 (+ 619us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.294285 (+ 539us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.294316 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.294319 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.294345 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.295850 (+ 1505us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.296345 (+ 495us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.296348 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.296353 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.296404 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.296661 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.296664 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.296666 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.296820 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.296928 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.297179 (+ 251us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.297284 (+ 105us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260429 20:00:36.297469 2606 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.289809 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37548)
0429 20:00:36.290112 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.290129 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.290258 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.290603 (+ 345us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.290606 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.290833 (+ 227us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.291265 (+ 432us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.291275 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.292376 (+ 1101us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.292380 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.292995 (+ 615us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.293003 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.293145 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.293793 (+ 648us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.293828 (+ 35us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.294158 (+ 330us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.296011 (+ 1853us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.296016 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.296018 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.296209 (+ 191us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.296507 (+ 298us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.296510 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.296512 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.296567 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.296917 (+ 350us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.296920 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.297017 (+ 97us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.297125 (+ 108us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.297276 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":182,"threads_started":1}
I20260429 20:00:36.298373 2043 tablet_copy_service.cc:140] P 4a284ad344d34d8c9e4cf992f4bb42d4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4ee5c1d0c7ac4d118e868a8aa6a57ce3 ({username='oryx', principal='oryx/127.27.17.253@KRBTEST.COM'} at 127.0.0.1:37548)
I20260429 20:00:36.298556 2043 tablet_copy_service.cc:161] P 4a284ad344d34d8c9e4cf992f4bb42d4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4ee5c1d0c7ac4d118e868a8aa6a57ce3 at {username='oryx', principal='oryx/127.27.17.253@KRBTEST.COM'} at 127.0.0.1:37548: session id = 4ee5c1d0c7ac4d118e868a8aa6a57ce3-00000000000000000000000000000000
I20260429 20:00:36.299943 2043 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Tablet Copy: opened 0 blocks and 1 log segments
WARNING: no policy specified for HTTP/127.27.17.252@KRBTEST.COM; defaulting to no policy
I20260429 20:00:36.301920 2495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.252@KRBTEST.COM" created.
I20260429 20:00:36.304695 2495 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: tablet copy: Starting download of 0 data blocks...
I20260429 20:00:36.304934 2495 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: tablet copy: Starting download of 1 WAL segments...
I20260429 20:00:36.307890 2495 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260429 20:00:36.310535 2043 tablet_copy_service.cc:342] P 4a284ad344d34d8c9e4cf992f4bb42d4: Request end of tablet copy session 4ee5c1d0c7ac4d118e868a8aa6a57ce3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.17.253@KRBTEST.COM'} at 127.0.0.1:37548
I20260429 20:00:36.310698 2043 tablet_copy_service.cc:434] P 4a284ad344d34d8c9e4cf992f4bb42d4: ending tablet copy session 4ee5c1d0c7ac4d118e868a8aa6a57ce3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4ee5c1d0c7ac4d118e868a8aa6a57ce3
W20260429 20:00:36.317154 2495 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260429 20:00:36.319847 2495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:36.321761 2609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.321848 2610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:36.322398 2495 server_base.cc:1061] running on GCE node
W20260429 20:00:36.323055 2612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:36.323339 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:36.323714 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:36.324870 2495 hybrid_clock.cc:648] HybridClock initialized: now 1777492836324852 us; error 33 us; skew 500 ppm
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:36.326187 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.252:33367
--webserver_interface=127.27.17.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:36.327163 2495 init.cc:377] Logged in from keytab as oryx/127.27.17.253@KRBTEST.COM (short username oryx)
I20260429 20:00:36.327708 2495 webserver.cc:492] Webserver started at http://127.27.17.253:44681/ using document root <none> and password file <none>
I20260429 20:00:36.327906 2495 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:36.327950 2495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:36.329310 2495 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:36.330549 2620 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:36.330868 2495 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:36.330999 2495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "48879457bbadb3b2361882d607cd1471"
server_key_iv: "b13297a3423cdac17b0e4882faf2f757"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.331095 2495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:36.361847 2495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:36.363986 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:36.369992 2495 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.253:43877
I20260429 20:00:36.370589 2495 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-1/data/info.pb
I20260429 20:00:36.372316 2673 sys_catalog.cc:263] Verifying existing consensus state
I20260429 20:00:36.373173 2673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Bootstrap starting.
I20260429 20:00:36.375149 2672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.253:43877 every 8 connection(s)
I20260429 20:00:36.381544 2673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 20:00:36.381888 2673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Bootstrap complete.
I20260429 20:00:36.382493 2673 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } }
I20260429 20:00:36.382633 2673 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4ee5c1d0c7ac4d118e868a8aa6a57ce3, State: Initialized, Role: LEARNER
I20260429 20:00:36.382750 2673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } }
I20260429 20:00:36.383155 2676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.383314 2676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: LEARNER
I20260429 20:00:36.383343 2673 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:36.386365 2687 catalog_manager.cc:1269] Loaded cluster ID: dae3886cdad348fc95dcb8ee81b09b07
I20260429 20:00:36.387110 2687 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: loading cluster ID for follower catalog manager: success
I20260429 20:00:36.388773 2687 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: acquiring CA information for follower catalog manager: success
I20260429 20:00:36.389839 2687 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 20:00:36.425370 2216 heartbeater.cc:499] Master 127.27.17.254:45045 was elected leader, sending a full tablet report...
W20260429 20:00:36.468837 2617 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:36.469141 2617 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:36.469269 2617 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:36.472739 2617 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:36.472841 2617 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:36.472882 2617 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:36.472945 2617 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:36.473003 2617 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:36.477877 2617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.252:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.17.252
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:36.479050 2617 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:36.479923 2617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:36.485774 2692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.485780 2695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.485796 2693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:36.486840 2617 server_base.cc:1061] running on GCE node
I20260429 20:00:36.487404 2617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:36.488389 2617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:36.489588 2617 hybrid_clock.cc:648] HybridClock initialized: now 1777492836489558 us; error 45 us; skew 500 ppm
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:36.492755 2617 init.cc:377] Logged in from keytab as oryx/127.27.17.252@KRBTEST.COM (short username oryx)
I20260429 20:00:36.493906 2617 webserver.cc:492] Webserver started at http://127.27.17.252:36249/ using document root <none> and password file <none>
I20260429 20:00:36.494858 2617 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:36.495050 2617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:36.496340 2617 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:36.499102 2617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/instance:
uuid: "49609ff57952474f8fdb3b87bf50a50a"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "61f6b8591d982a08a61ec66f97826803"
server_key_iv: "6d552b752c33a0d0cfd8254a7f038c84"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.499682 2617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal/instance:
uuid: "49609ff57952474f8fdb3b87bf50a50a"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "61f6b8591d982a08a61ec66f97826803"
server_key_iv: "6d552b752c33a0d0cfd8254a7f038c84"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.503376 2617 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:00:36.505671 2702 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:36.506817 2617 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:36.507018 2617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
uuid: "49609ff57952474f8fdb3b87bf50a50a"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "61f6b8591d982a08a61ec66f97826803"
server_key_iv: "6d552b752c33a0d0cfd8254a7f038c84"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.507138 2617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:36.532181 2617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:36.535815 2617 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:36.536029 2617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:36.544310 2617 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.252:33367
I20260429 20:00:36.544307 2754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.252:33367 every 8 connection(s)
I20260429 20:00:36.545346 2617 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/info.pb
I20260429 20:00:36.547202 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 2617
I20260429 20:00:36.548561 2755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:36.553187 2755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.17.252@KRBTEST.COM
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.252@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:36.562572 2757 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.548667 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.27.17.252:33367)
0429 20:00:36.549123 (+ 456us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.549132 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.549167 (+ 35us) server_negotiation.cc:408] Connection header received
0429 20:00:36.549960 (+ 793us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.549982 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.550286 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.550586 (+ 300us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.551890 (+ 1304us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.552724 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.553868 (+ 1144us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.554321 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.556873 (+ 2552us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.556925 (+ 52us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.556936 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.556970 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.559613 (+ 2643us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.560131 (+ 518us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.560139 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.560147 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.560252 (+ 105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.560655 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.560658 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.560660 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.561072 (+ 412us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.561336 (+ 264us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.561634 (+ 298us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.561899 (+ 265us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":150,"threads_started":1}
I20260429 20:00:36.564786 2758 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.555524 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37558)
0429 20:00:36.555991 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.556030 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.556168 (+ 138us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.556565 (+ 397us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.556571 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.557159 (+ 588us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.557500 (+ 341us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.557512 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.558435 (+ 923us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.558439 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.559106 (+ 667us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.559116 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.559248 (+ 132us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.559744 (+ 496us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.559762 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.561349 (+ 1587us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.563231 (+ 1882us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.563238 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.563241 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.563442 (+ 201us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.563751 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.563754 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.563757 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.563815 (+ 58us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.564206 (+ 391us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.564211 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.564349 (+ 138us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.564489 (+ 140us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.564628 (+ 139us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":216,"threads_started":1}
I20260429 20:00:36.565008 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.555651 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37558 (local address 127.27.17.254:45045)
0429 20:00:36.555832 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.555837 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.556122 (+ 285us) server_negotiation.cc:408] Connection header received
0429 20:00:36.556281 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.556288 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.556355 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.556448 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.557676 (+ 1228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.558304 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.559243 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.559440 (+ 197us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.561499 (+ 2059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.561520 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.561522 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.561552 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.563096 (+ 1544us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.563553 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.563557 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.563559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.563619 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.563936 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.563939 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.563941 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.564094 (+ 153us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.564187 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.564606 (+ 419us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.564852 (+ 246us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260429 20:00:36.566071 2755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.252@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
I20260429 20:00:36.575858 2758 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.566702 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34216)
0429 20:00:36.566845 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.566858 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.566981 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.567383 (+ 402us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.567386 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.567649 (+ 263us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.567841 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.567847 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.568833 (+ 986us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.568837 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.569262 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.569269 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.569480 (+ 211us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.570038 (+ 558us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.570052 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.571712 (+ 1660us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.574102 (+ 2390us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.574105 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.574107 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.574348 (+ 241us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.574684 (+ 336us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.574687 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.574688 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.574745 (+ 57us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.575262 (+ 517us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.575265 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.575358 (+ 93us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.575574 (+ 216us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.575716 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260429 20:00:36.576041 2759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.566825 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.27.17.253:43877)
0429 20:00:36.567138 (+ 313us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.567142 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.567154 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:36.567190 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.567192 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.567237 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.567322 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.567982 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.568703 (+ 721us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.569518 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.569718 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.571887 (+ 2169us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.571909 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.571912 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.571939 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.573962 (+ 2023us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.574502 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.574506 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.574508 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.574563 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.574867 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.574871 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.574872 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.575122 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.575222 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.575733 (+ 511us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.575856 (+ 123us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":163,"threads_started":1}
I20260429 20:00:36.577006 2755 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } has no permanent_uuid. Determining permanent_uuid...
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.252@KRBTEST.COM for oryx/127.27.17.252@KRBTEST.COM
I20260429 20:00:36.586618 2758 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.577568 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37416)
0429 20:00:36.577803 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.577818 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.577895 (+ 77us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.578395 (+ 500us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.578398 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.578628 (+ 230us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.578821 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.578827 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.579748 (+ 921us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.579751 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.580108 (+ 357us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.580114 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.580217 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.580871 (+ 654us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.580885 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.582258 (+ 1373us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.584907 (+ 2649us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.584910 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.584912 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.585257 (+ 345us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.585673 (+ 416us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.585676 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.585678 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.585726 (+ 48us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.586180 (+ 454us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.586183 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.586269 (+ 86us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.586363 (+ 94us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.586482 (+ 119us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260429 20:00:36.586778 2757 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.577891 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.27.17.252:33367)
0429 20:00:36.578090 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.578093 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.578105 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:36.578134 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.578137 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.578176 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.578247 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.579049 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.579592 (+ 543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.580276 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.580458 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.582444 (+ 1986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.582464 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.582467 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.582492 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.584750 (+ 2258us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.585451 (+ 701us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.585458 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.585460 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.585510 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.585848 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.585852 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.585853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.586023 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.586120 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.586495 (+ 375us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.586594 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260429 20:00:36.590319 2755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Bootstrap starting.
I20260429 20:00:36.593209 2755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:36.593988 2755 log.cc:826] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:36.596251 2755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: No bootstrap required, opened a new log
I20260429 20:00:36.599174 2755 raft_consensus.cc:359] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } }
I20260429 20:00:36.599411 2755 raft_consensus.cc:385] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:36.599489 2755 raft_consensus.cc:740] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49609ff57952474f8fdb3b87bf50a50a, State: Initialized, Role: FOLLOWER
I20260429 20:00:36.600016 2755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } }
I20260429 20:00:36.600729 2761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } } }
I20260429 20:00:36.600847 2761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:36.601317 2755 sys_catalog.cc:565] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: configured and running, proceeding with master startup.
W20260429 20:00:36.606383 2772 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:00:36.606492 2772 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:00:36.611191 2617 master_runner.cc:428] Detected that this master 49609ff57952474f8fdb3b87bf50a50a is joining an existing cluster
I20260429 20:00:36.611356 2617 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.17.252:33367
I20260429 20:00:36.621291 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.617711 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37570)
0429 20:00:36.617868 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.617882 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.617955 (+ 73us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.618931 (+ 976us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.618938 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.618961 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:36.619172 (+ 211us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.619178 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.620199 (+ 1021us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.620205 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.620809 (+ 604us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.620816 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.620955 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.621024 (+ 69us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.621081 (+ 57us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.621139 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260429 20:00:36.621628 2780 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.617854 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.27.17.254:45045)
0429 20:00:36.618325 (+ 471us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.618329 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.618645 (+ 316us) server_negotiation.cc:408] Connection header received
0429 20:00:36.618713 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.618716 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.618757 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.618880 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:36.619399 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.620060 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.620932 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.621372 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.621434 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.621491 (+ 57us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.621539 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"spinlock_wait_cycles":1920,"thread_start_us":102,"threads_started":1}
I20260429 20:00:36.625852 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.623147 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34232)
0429 20:00:36.623427 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.623445 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.623532 (+ 87us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.623805 (+ 273us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.623809 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.623827 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:36.623999 (+ 172us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.624003 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.624737 (+ 734us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.624742 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.625336 (+ 594us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.625342 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.625561 (+ 219us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.625582 (+ 21us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.625661 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.625722 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260429 20:00:36.626360 2759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.623338 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34232 (local address 127.27.17.253:43877)
0429 20:00:36.623530 (+ 192us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.623534 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.623574 (+ 40us) server_negotiation.cc:408] Connection header received
0429 20:00:36.623638 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.623641 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.623695 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.623794 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:36.624110 (+ 316us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.624620 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.625525 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.626021 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.626173 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.626219 (+ 46us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.626272 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260429 20:00:36.643226 2761 raft_consensus.cc:493] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:36.643469 2761 raft_consensus.cc:515] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } }
I20260429 20:00:36.644537 2761 leader_election.cc:290] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253:43877)
I20260429 20:00:36.644901 2647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49609ff57952474f8fdb3b87bf50a50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" is_pre_election: true
I20260429 20:00:36.645066 2647 raft_consensus.cc:1813] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Handling vote request from an unknown peer 49609ff57952474f8fdb3b87bf50a50a
I20260429 20:00:36.645032 2033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49609ff57952474f8fdb3b87bf50a50a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" is_pre_election: true
I20260429 20:00:36.645190 2647 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 49609ff57952474f8fdb3b87bf50a50a for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260429 20:00:36.645740 2706 leader_election.cc:304] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49609ff57952474f8fdb3b87bf50a50a; no voters: 4a284ad344d34d8c9e4cf992f4bb42d4, 4ee5c1d0c7ac4d118e868a8aa6a57ce3
I20260429 20:00:36.646049 2761 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:36.648146 2761 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260429 20:00:36.659833 2352 heartbeater.cc:499] Master 127.27.17.254:45045 was elected leader, sending a full tablet report...
I20260429 20:00:36.660091 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.613912 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37562 (local address 127.27.17.254:45045)
0429 20:00:36.614074 (+ 162us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.614080 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.615791 (+ 1711us) server_negotiation.cc:408] Connection header received
0429 20:00:36.615902 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.615905 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.615963 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.616048 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.647813 (+ 31765us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.648581 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.649607 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.649825 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.655136 (+ 5311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.655158 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.655163 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.655201 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.658180 (+ 2979us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.658808 (+ 628us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.658813 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.658815 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.658869 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659147 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.659150 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.659153 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.659347 (+ 194us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.659443 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.659801 (+ 358us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.659921 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260429 20:00:36.660794 2779 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.614886 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37562)
0429 20:00:36.615689 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.615704 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.615808 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.616173 (+ 365us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.616177 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.647194 (+ 31017us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.647644 (+ 450us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.647658 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.648766 (+ 1108us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.648770 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.649354 (+ 584us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.649361 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.649961 (+ 600us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.654485 (+ 4524us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.654513 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.654977 (+ 464us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.658328 (+ 3351us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.658332 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.658335 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.658685 (+ 350us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.658996 (+ 311us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.658999 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.659001 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.659053 (+ 52us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.659507 (+ 454us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.659510 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.659590 (+ 80us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.660241 (+ 651us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.660384 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"spinlock_wait_cycles":11648,"thread_start_us":138,"threads_started":1}
I20260429 20:00:36.662003 2759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.647792 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.27.17.253:43877)
0429 20:00:36.647922 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.647926 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.648571 (+ 645us) server_negotiation.cc:408] Connection header received
0429 20:00:36.648747 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.648751 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.648801 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.648852 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.649703 (+ 851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.650850 (+ 1147us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.651809 (+ 959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.651950 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.656364 (+ 4414us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.656380 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.656386 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.656411 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.658864 (+ 2453us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659344 (+ 480us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.659347 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.659350 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.659397 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659770 (+ 373us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.659774 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.659776 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.659942 (+ 166us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.660012 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.660651 (+ 639us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.660792 (+ 141us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 20:00:36.662058 2757 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.614508 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.27.17.252:33367)
0429 20:00:36.614610 (+ 102us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.614613 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.614625 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:36.614656 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.614659 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.614695 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.614797 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.648515 (+ 33718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.649138 (+ 623us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.650034 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.650230 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.656057 (+ 5827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.656074 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.656077 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.656099 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.658161 (+ 2062us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.658934 (+ 773us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.658939 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.658941 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.658988 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659226 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.659229 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.659231 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.659453 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.659637 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.661797 (+ 2160us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.661914 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260429 20:00:36.662448 2778 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.613909 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37430)
0429 20:00:36.614341 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.614358 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.614467 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.614806 (+ 339us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.614809 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.648083 (+ 33274us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.648387 (+ 304us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.648396 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.649252 (+ 856us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.649255 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.649901 (+ 646us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.649912 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.650339 (+ 427us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.654941 (+ 4602us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.654962 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.655355 (+ 393us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.658325 (+ 2970us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.658330 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.658332 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.658783 (+ 451us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.659086 (+ 303us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659089 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.659091 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.659138 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.661530 (+ 2392us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.661533 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.661622 (+ 89us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.662178 (+ 556us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.662296 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":234,"threads_started":1}
I20260429 20:00:36.662592 2781 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.647247 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34242)
0429 20:00:36.648473 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.648499 (+ 26us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.648639 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.649048 (+ 409us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.649054 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.649276 (+ 222us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.649561 (+ 285us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.649569 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.651039 (+ 1470us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.651043 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.651660 (+ 617us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.651668 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.651806 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.654351 (+ 2545us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.654371 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.655917 (+ 1546us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.659019 (+ 3102us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659024 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.659029 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.659242 (+ 213us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.659587 (+ 345us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.659593 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.659598 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.659661 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.660083 (+ 422us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.660089 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.660270 (+ 181us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.662250 (+ 1980us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.662401 (+ 151us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1148,"thread_start_us":128,"threads_started":1}
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492834, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.254@KRBTEST.COM for oryx/127.27.17.252@KRBTEST.COM
I20260429 20:00:36.674610 2757 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.665839 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37444 (local address 127.27.17.252:33367)
0429 20:00:36.665978 (+ 139us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.665981 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.666012 (+ 31us) server_negotiation.cc:408] Connection header received
0429 20:00:36.666075 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.666077 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.666121 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.666182 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.666860 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.667379 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.668205 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.668371 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.670518 (+ 2147us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.670541 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.670544 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.670575 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.672540 (+ 1965us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.673192 (+ 652us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.673196 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.673198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.673251 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.673760 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.673764 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.673766 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.673949 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.674061 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.674365 (+ 304us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.674475 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 20:00:36.674610 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.665677 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37444)
0429 20:00:36.665855 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.665870 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.665962 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.666223 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.666227 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.666442 (+ 215us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.666717 (+ 275us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.666725 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.667519 (+ 794us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.667523 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.668080 (+ 557us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.668087 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.668207 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.668717 (+ 510us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.668735 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.670344 (+ 1609us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.672808 (+ 2464us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.672817 (+ 9us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.672820 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.673025 (+ 205us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.673359 (+ 334us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.673363 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.673365 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.673538 (+ 173us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.674052 (+ 514us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.674055 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.674157 (+ 102us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.674267 (+ 110us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.674425 (+ 158us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260429 20:00:36.675634 2022 catalog_manager.cc:7158] Initiating ChangeConfig request to add master 127.27.17.252:33367: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a"
member_type: NON_VOTER
last_known_addr {
host: "127.27.17.252"
port: 33367
}
attrs {
promote: true
}
}
dest_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4"
cas_config_opid_index: 5
I20260429 20:00:36.676419 2022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } }
I20260429 20:00:36.677515 2592 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.677893 2647 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Refusing update from remote peer 4a284ad344d34d8c9e4cf992f4bb42d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260429 20:00:36.678136 2729 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Refusing update from remote peer 4a284ad344d34d8c9e4cf992f4bb42d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260429 20:00:36.678328 2595 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260429 20:00:36.678634 2593 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:36.678995 2592 catalog_manager.cc:7162] Successfully completed master ChangeConfig request to add master 127.27.17.252:33367
I20260429 20:00:36.679207 2594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.679309 2594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:36.680737 2647 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.682734 2676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a284ad344d34d8c9e4cf992f4bb42d4. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.682869 2676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: LEARNER
I20260429 20:00:36.683000 2676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.683087 2676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: LEARNER
I20260429 20:00:36.683329 2617 master.cc:562] Master@127.27.17.252:33367 shutting down...
I20260429 20:00:36.683629 2592 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: attempting to promote NON_VOTER 4ee5c1d0c7ac4d118e868a8aa6a57ce3 to VOTER
I20260429 20:00:36.684016 2592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } }
I20260429 20:00:36.684160 2729 raft_consensus.cc:872] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } }
W20260429 20:00:36.684273 2729 raft_consensus.cc:1579] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } }
I20260429 20:00:36.685928 2647 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 LEARNER]: Refusing update from remote peer 4a284ad344d34d8c9e4cf992f4bb42d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260429 20:00:36.686349 2593 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260429 20:00:36.686728 2008 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 -> Peer 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252:33367): Couldn't send request to peer 49609ff57952474f8fdb3b87bf50a50a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260429 20:00:36.686833 2729 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.688648 2593 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.689630 2761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a284ad344d34d8c9e4cf992f4bb42d4. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.689734 2761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260429 20:00:36.689875 2761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.689935 2761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260429 20:00:36.689965 2786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: NON_VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: true } } }
I20260429 20:00:36.690197 2786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260429 20:00:36.690374 2592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.690486 2592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:36.690407 2593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.690598 2593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:36.691443 2647 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.692631 2783 mvcc.cc:204] Tried to move back new op lower bound from 7280610652642713600 to 7280610652452691968. Current Snapshot: MvccSnapshot[applied={T|T < 7280610652491370496}]
I20260429 20:00:36.692945 2784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.693063 2784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:36.695613 2617 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260429 20:00:36.695935 2617 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260429 20:00:36.696030 2617 tablet_replica.cc:333] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: stopping tablet replica
I20260429 20:00:36.711876 2617 master.cc:584] Master@127.27.17.252:33367 shutdown complete.
I20260429 20:00:36.712076 2617 master_runner.cc:305] Clearing existing system tablet
I20260429 20:00:36.712574 2617 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:36.712677 2617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:36.712949 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.712602 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37458)
0429 20:00:36.712764 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.712844 (+ 80us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.252:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260429 20:00:36.713258 2022 master.cc:680] Network error: unable to get registration information for peer 49609ff57952474f8fdb3b87bf50a50a: Client connection negotiation failed: client connection to 127.27.17.252:33367: connect: Connection refused (error 111)
I20260429 20:00:36.714733 2617 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:36.715771 2792 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:36.715993 2617 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:36.716130 2617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260429 20:00:36.716171 2617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
uuid: "49609ff57952474f8fdb3b87bf50a50a"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "61f6b8591d982a08a61ec66f97826803"
server_key_iv: "6d552b752c33a0d0cfd8254a7f038c84"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.717648 2617 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:36.719897 2617 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260429 20:00:36.719987 2617 log.cc:1199] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260429 20:00:36.720358 2617 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Deleting consensus metadata
I20260429 20:00:36.720551 2617 master_runner.cc:315] Copying system tablet from 127.27.17.254:45045
I20260429 20:00:36.721763 2617 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: tablet copy: Beginning tablet copy session from remote peer at address 127.27.17.254:45045
I20260429 20:00:36.730139 2798 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.722347 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37574)
0429 20:00:36.722667 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.722680 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.722793 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.723119 (+ 326us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.723122 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.723295 (+ 173us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:36.723524 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.723530 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.724588 (+ 1058us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.724592 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.725012 (+ 420us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.725019 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.725122 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.726274 (+ 1152us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:36.726288 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:36.726586 (+ 298us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:36.728636 (+ 2050us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.728642 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.728644 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.728841 (+ 197us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.729094 (+ 253us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:36.729097 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:36.729099 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:36.729152 (+ 53us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:36.729704 (+ 552us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:36.729707 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:36.729793 (+ 86us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.729887 (+ 94us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.729999 (+ 112us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260429 20:00:36.730263 2062 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.722435 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.27.17.254:45045)
0429 20:00:36.722623 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.722628 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.722757 (+ 129us) server_negotiation.cc:408] Connection header received
0429 20:00:36.722890 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.722911 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.722975 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.723073 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:36.723648 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.724449 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.725131 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.725342 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.726712 (+ 1370us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:36.726734 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:36.726738 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:36.726764 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:36.728498 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.728942 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.728946 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.728948 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.729004 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:36.729245 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:36.729248 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:36.729250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:36.729435 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:36.729674 (+ 239us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.729973 (+ 299us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.730099 (+ 126us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 20:00:36.730724 2043 tablet_copy_service.cc:140] P 4a284ad344d34d8c9e4cf992f4bb42d4: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 49609ff57952474f8fdb3b87bf50a50a ({username='oryx', principal='oryx/127.27.17.252@KRBTEST.COM'} at 127.0.0.1:37574)
I20260429 20:00:36.730830 2043 tablet_copy_service.cc:161] P 4a284ad344d34d8c9e4cf992f4bb42d4: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 49609ff57952474f8fdb3b87bf50a50a at {username='oryx', principal='oryx/127.27.17.252@KRBTEST.COM'} at 127.0.0.1:37574: session id = 49609ff57952474f8fdb3b87bf50a50a-00000000000000000000000000000000
I20260429 20:00:36.731891 2043 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Tablet Copy: opened 0 blocks and 1 log segments
I20260429 20:00:36.733218 2617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260429 20:00:36.735675 2617 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: tablet copy: Starting download of 0 data blocks...
I20260429 20:00:36.735872 2617 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: tablet copy: Starting download of 1 WAL segments...
I20260429 20:00:36.737809 2617 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260429 20:00:36.739622 2043 tablet_copy_service.cc:342] P 4a284ad344d34d8c9e4cf992f4bb42d4: Request end of tablet copy session 49609ff57952474f8fdb3b87bf50a50a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.17.252@KRBTEST.COM'} at 127.0.0.1:37574
I20260429 20:00:36.739727 2043 tablet_copy_service.cc:434] P 4a284ad344d34d8c9e4cf992f4bb42d4: ending tablet copy session 49609ff57952474f8fdb3b87bf50a50a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 49609ff57952474f8fdb3b87bf50a50a
W20260429 20:00:36.744364 2617 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260429 20:00:36.746948 2617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:36.748813 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.748823 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:36.748865 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:36.749341 2617 server_base.cc:1061] running on GCE node
I20260429 20:00:36.749553 2617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:36.749878 2617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:36.750979 2617 hybrid_clock.cc:648] HybridClock initialized: now 1777492836750967 us; error 26 us; skew 500 ppm
Apr 29 20:00:36 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492836, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:36.752801 2617 init.cc:377] Logged in from keytab as oryx/127.27.17.252@KRBTEST.COM (short username oryx)
I20260429 20:00:36.753408 2617 webserver.cc:492] Webserver started at http://127.27.17.252:38857/ using document root <none> and password file <none>
I20260429 20:00:36.753696 2617 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:36.753746 2617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:36.754998 2617 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:36.755954 2809 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:36.756187 2617 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:36.756254 2617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
uuid: "49609ff57952474f8fdb3b87bf50a50a"
format_stamp: "Formatted at 2026-04-29 20:00:36 on dist-test-slave-01nr"
server_key: "61f6b8591d982a08a61ec66f97826803"
server_key_iv: "6d552b752c33a0d0cfd8254a7f038c84"
server_key_version: "encryptionkey@0"
I20260429 20:00:36.756354 2617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:36.781791 2617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:36.783658 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.783199 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37474)
0429 20:00:36.783350 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.783402 (+ 52us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.252:33367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260429 20:00:36.783931 2022 master.cc:680] Network error: unable to get registration information for peer 49609ff57952474f8fdb3b87bf50a50a: Client connection negotiation failed: client connection to 127.27.17.252:33367: connect: Connection refused (error 111)
I20260429 20:00:36.784556 2617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:36.789880 2617 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.252:33367
I20260429 20:00:36.789891 2861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.252:33367 every 8 connection(s)
I20260429 20:00:36.790612 2617 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/master-2/data/info.pb
I20260429 20:00:36.792204 2862 sys_catalog.cc:263] Verifying existing consensus state
I20260429 20:00:36.793258 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Bootstrap starting.
I20260429 20:00:36.801182 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 20:00:36.801550 2862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: Bootstrap complete.
I20260429 20:00:36.802139 2862 raft_consensus.cc:359] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } }
I20260429 20:00:36.802253 2862 raft_consensus.cc:740] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49609ff57952474f8fdb3b87bf50a50a, State: Initialized, Role: LEARNER
I20260429 20:00:36.802376 2862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } }
I20260429 20:00:36.802835 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:36.802923 2862 sys_catalog.cc:565] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:36.803027 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: LEARNER
I20260429 20:00:36.806495 2876 catalog_manager.cc:1269] Loaded cluster ID: dae3886cdad348fc95dcb8ee81b09b07
I20260429 20:00:36.806576 2876 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: loading cluster ID for follower catalog manager: success
I20260429 20:00:36.808542 2876 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: acquiring CA information for follower catalog manager: success
I20260429 20:00:36.809738 2876 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260429 20:00:36.920516 2588 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.916992 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37480)
0429 20:00:36.917185 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:36.917203 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:36.917343 (+ 140us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:36.917973 (+ 630us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:36.917976 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:36.918002 (+ 26us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:36.918188 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.918193 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.919337 (+ 1144us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.919341 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:36.920054 (+ 713us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:36.920062 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.920181 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.920197 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 20:00:36.920240 (+ 43us) client_negotiation.cc:241] Negotiation successful
0429 20:00:36.920292 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260429 20:00:36.921046 2877 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:36.917185 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.27.17.252:33367)
0429 20:00:36.917662 (+ 477us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:36.917666 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:36.917680 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:36.917726 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:36.917728 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:36.917780 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:36.917878 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:36.918366 (+ 488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.919181 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:36.920205 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:36.920622 (+ 417us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:36.920804 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:36.920870 (+ 66us) server_negotiation.cc:300] Negotiation successful
0429 20:00:36.920932 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":119,"threads_started":1}
I20260429 20:00:36.921037 2488 heartbeater.cc:499] Master 127.27.17.254:45045 was elected leader, sending a full tablet report...
I20260429 20:00:37.205118 2829 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260429 20:00:37.205534 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a284ad344d34d8c9e4cf992f4bb42d4. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:37.205682 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: LEARNER
I20260429 20:00:37.664896 2593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: NON_VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: true } } }
I20260429 20:00:37.665086 2593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:37.666473 2878 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: attempting to promote NON_VOTER 49609ff57952474f8fdb3b87bf50a50a to VOTER
I20260429 20:00:37.666962 2878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:37.668468 2647 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Refusing update from remote peer 4a284ad344d34d8c9e4cf992f4bb42d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260429 20:00:37.668535 2829 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 LEARNER]: Refusing update from remote peer 4a284ad344d34d8c9e4cf992f4bb42d4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260429 20:00:37.668996 2880 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260429 20:00:37.669291 2878 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260429 20:00:37.671275 2878 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.671751 2647 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.671877 2829 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.672852 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.672986 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:37.673000 2593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.673132 2593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:37.673198 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.673310 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:37.673130 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.673666 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:37.673854 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:37.673962 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:37.709246 2885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.702345 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.27.17.253:43877)
0429 20:00:37.702642 (+ 297us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.702646 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.702660 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:37.702694 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.702697 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.702738 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.702822 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:37.703754 (+ 932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.704325 (+ 571us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.704992 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.705289 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.706317 (+ 1028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:37.706337 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:37.706339 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:37.706366 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:37.707879 (+ 1513us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.708243 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.708246 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.708247 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.708291 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.708494 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.708496 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.708497 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.708638 (+ 141us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:37.708736 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.708915 (+ 179us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.709016 (+ 101us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
I20260429 20:00:37.712818 2886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.710079 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37582)
0429 20:00:37.710318 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.710333 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.710422 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.710771 (+ 349us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.710773 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.710787 (+ 14us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:37.711016 (+ 229us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.711021 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.711840 (+ 819us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.711843 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.712494 (+ 651us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.712502 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.712607 (+ 105us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.712620 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.712658 (+ 38us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.712702 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260429 20:00:37.713376 2887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.710216 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.27.17.254:45045)
0429 20:00:37.710576 (+ 360us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.710580 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.710592 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:37.710635 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.710638 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.710679 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.710753 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:37.711147 (+ 394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.711730 (+ 583us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.712644 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.713154 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.713188 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.713232 (+ 44us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.713273 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":155,"threads_started":1}
I20260429 20:00:37.717229 2886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.714658 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34264)
0429 20:00:37.714796 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.714808 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.714870 (+ 62us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.715204 (+ 334us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.715208 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.715231 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:37.715416 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.715421 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.716372 (+ 951us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.716375 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.716929 (+ 554us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.716936 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.717036 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.717048 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.717085 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.717127 (+ 42us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260429 20:00:37.717689 2885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.714807 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.27.17.253:43877)
0429 20:00:37.714970 (+ 163us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.714973 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.714984 (+ 11us) server_negotiation.cc:408] Connection header received
0429 20:00:37.715019 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.715021 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.715056 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.715119 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:37.715573 (+ 454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.716234 (+ 661us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.717058 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.717476 (+ 418us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.717522 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.717563 (+ 41us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.717607 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 20:00:37.721261 2886 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.718793 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37484)
0429 20:00:37.718935 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.718956 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.719016 (+ 60us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.719446 (+ 430us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.719449 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.719467 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:37.719621 (+ 154us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.719625 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.720357 (+ 732us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.720361 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.720911 (+ 550us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.720917 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.721041 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.721054 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.721103 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.721149 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260429 20:00:37.721732 2888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.718977 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.27.17.252:33367)
0429 20:00:37.719243 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.719247 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.719258 (+ 11us) server_negotiation.cc:408] Connection header received
0429 20:00:37.719298 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.719301 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.719343 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.719438 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:37.719729 (+ 291us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.720220 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.721029 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.721503 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.721547 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.721601 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.721640 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":99,"threads_started":1}
I20260429 20:00:37.729882 2888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.723621 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37498 (local address 127.27.17.252:33367)
0429 20:00:37.723731 (+ 110us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.723735 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.723746 (+ 11us) server_negotiation.cc:408] Connection header received
0429 20:00:37.723810 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.723812 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.723847 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.723947 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:37.724694 (+ 747us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.725149 (+ 455us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.725748 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.725923 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.727006 (+ 1083us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:37.727020 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:37.727023 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:37.727044 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:37.728687 (+ 1643us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.729008 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.729011 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.729012 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.729048 (+ 36us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.729251 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.729254 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.729255 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.729376 (+ 121us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:37.729476 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.729673 (+ 197us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.729765 (+ 92us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260429 20:00:37.733125 2889 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.730657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:37584)
0429 20:00:37.730876 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.730888 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.731009 (+ 121us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.731266 (+ 257us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.731269 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.731287 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:37.731465 (+ 178us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.731486 (+ 21us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.732189 (+ 703us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.732192 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.732811 (+ 619us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.732819 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.732920 (+ 101us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.732931 (+ 11us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.732969 (+ 38us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.733005 (+ 36us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260429 20:00:37.733603 2887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.730788 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37584 (local address 127.27.17.254:45045)
0429 20:00:37.730957 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.730962 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.730982 (+ 20us) server_negotiation.cc:408] Connection header received
0429 20:00:37.731097 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.731099 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.731142 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.731205 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:37.731591 (+ 386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.732072 (+ 481us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.732932 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.733385 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.733423 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.733465 (+ 42us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.733511 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 20:00:37.737442 2889 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.735069 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.0.0.1:34276)
0429 20:00:37.735218 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.735232 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.735355 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.735616 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.735619 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.735637 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:37.735794 (+ 157us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.735798 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.736503 (+ 705us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.736506 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.737077 (+ 571us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.737083 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.737207 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.737221 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.737270 (+ 49us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.737321 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260429 20:00:37.737885 2885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.735227 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.27.17.253:43877)
0429 20:00:37.735402 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.735409 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.735424 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:37.735476 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.735479 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.735521 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.735607 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:37.735900 (+ 293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.736387 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.737193 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.737667 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.737701 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.737757 (+ 56us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.737798 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260429 20:00:37.741400 2889 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.739255 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.0.0.1:37502)
0429 20:00:37.739388 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.739404 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.739538 (+ 134us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.739772 (+ 234us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.739775 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.739793 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:37.739927 (+ 134us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.739931 (+ 4us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.740587 (+ 656us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.740590 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.741125 (+ 535us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.741131 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.741221 (+ 90us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.741232 (+ 11us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.741270 (+ 38us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.741308 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260429 20:00:37.741827 2888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.739362 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.27.17.252:33367)
0429 20:00:37.739446 (+ 84us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.739449 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.739460 (+ 11us) server_negotiation.cc:408] Connection header received
0429 20:00:37.739626 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.739628 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.739665 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.739729 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:37.740024 (+ 295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.740487 (+ 463us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.741232 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.741652 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.741679 (+ 27us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.741717 (+ 38us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.741752 (+ 35us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260429 20:00:37.743295 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 2087
I20260429 20:00:37.749617 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:34583
--local_ip_for_outbound_sockets=127.27.17.193
--tserver_master_addrs=127.27.17.254:45045
--webserver_port=42345
--webserver_interface=127.27.17.193
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:37.855567 2890 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:37.855834 2890 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:37.855942 2890 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:37.859503 2890 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:37.859599 2890 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:37.859714 2890 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:37.864192 2890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:34583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=42345
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:37.865355 2890 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:37.866261 2890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:37.873596 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:37.873628 2896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:37.873644 2895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:37.873739 2890 server_base.cc:1061] running on GCE node
I20260429 20:00:37.874436 2890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:37.875118 2890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:37.876298 2890 hybrid_clock.cc:648] HybridClock initialized: now 1777492837876262 us; error 50 us; skew 500 ppm
Apr 29 20:00:37 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492837, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:37.879236 2890 init.cc:377] Logged in from keytab as oryx/127.27.17.193@KRBTEST.COM (short username oryx)
I20260429 20:00:37.880343 2890 webserver.cc:492] Webserver started at http://127.27.17.193:42345/ using document root <none> and password file <none>
I20260429 20:00:37.880911 2890 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:37.881004 2890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:37.884593 2890 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:37.886449 2905 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:37.887663 2890 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:37.887760 2890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "dc242720f061411e98271e1a9ad950b1"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "752711b626a44d2727022337fb694896"
server_key_iv: "7625522bdd7c7af1d7eb433f6d4ebc61"
server_key_version: "encryptionkey@0"
I20260429 20:00:37.888180 2890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:37.901821 2890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:37.905017 2890 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:37.905239 2890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:37.905771 2890 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:37.906646 2890 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:37.906692 2890 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:37.906729 2890 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:37.906744 2890 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:37.917212 2890 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:34583
I20260429 20:00:37.917232 3018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:34583 every 8 connection(s)
I20260429 20:00:37.918365 2890 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:37.925729 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 2890
I20260429 20:00:37.925949 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 2223
Apr 29 20:00:37 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492837, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.252@KRBTEST.COM
Apr 29 20:00:37 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492837, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
I20260429 20:00:37.932806 2888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.920169 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:60925 (local address 127.27.17.252:33367)
0429 20:00:37.920319 (+ 150us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.920347 (+ 28us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.921355 (+ 1008us) server_negotiation.cc:408] Connection header received
0429 20:00:37.922249 (+ 894us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.922253 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.922311 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.922374 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:37.924104 (+ 1730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.924707 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.925449 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.925655 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.928334 (+ 2679us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:37.928353 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:37.928357 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:37.928391 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:37.930403 (+ 2012us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.930950 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.930957 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.930961 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.931017 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.931370 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.931377 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.931382 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.931564 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:37.931635 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.932455 (+ 820us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.932603 (+ 148us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260429 20:00:37.933643 2885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.921672 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:43993 (local address 127.27.17.253:43877)
0429 20:00:37.921799 (+ 127us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.921803 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.921917 (+ 114us) server_negotiation.cc:408] Connection header received
0429 20:00:37.922230 (+ 313us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.922242 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.922300 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.922375 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:37.924223 (+ 1848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.924990 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.925856 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.926080 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.929155 (+ 3075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:37.929179 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:37.929186 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:37.929219 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:37.931373 (+ 2154us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.931933 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.931940 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.931945 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.931997 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.932279 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.932285 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.932290 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.932519 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:37.932607 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.933326 (+ 719us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.933458 (+ 132us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Apr 29 20:00:37 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492837, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:37.934232 3024 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.920539 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.27.17.193:60925)
0429 20:00:37.921269 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.921281 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.922032 (+ 751us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.922551 (+ 519us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.922562 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.923272 (+ 710us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:37.923946 (+ 674us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.923960 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.924831 (+ 871us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.924834 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.925320 (+ 486us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.925334 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.925523 (+ 189us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.926140 (+ 617us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:37.926162 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:37.928118 (+ 1956us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:37.930547 (+ 2429us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:37.930554 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:37.930567 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:37.930812 (+ 245us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:37.931121 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:37.931128 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:37.931133 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:37.931264 (+ 131us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:37.931720 (+ 456us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:37.931726 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:37.932122 (+ 396us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.933064 (+ 942us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.933346 (+ 282us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":28,"spinlock_wait_cycles":13824,"thread_start_us":106,"threads_started":1}
I20260429 20:00:37.934376 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:44123
--local_ip_for_outbound_sockets=127.27.17.194
--tserver_master_addrs=127.27.17.254:45045
--webserver_port=43281
--webserver_interface=127.27.17.194
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 20:00:37.934955 3025 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.921557 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.27.17.193:43993)
0429 20:00:37.921823 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.921844 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.922047 (+ 203us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.922552 (+ 505us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.922560 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.923817 (+ 1257us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:37.924085 (+ 268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.924093 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.925112 (+ 1019us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.925115 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.925669 (+ 554us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.925680 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.925890 (+ 210us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.927360 (+ 1470us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:37.927383 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:37.929010 (+ 1627us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:37.931506 (+ 2496us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:37.931512 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:37.931517 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:37.931758 (+ 241us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:37.932104 (+ 346us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:37.932110 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:37.932115 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:37.932176 (+ 61us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:37.932714 (+ 538us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:37.932737 (+ 23us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:37.932826 (+ 89us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.934089 (+ 1263us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.934229 (+ 140us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
I20260429 20:00:37.935652 3019 heartbeater.cc:344] Connected to a master server at 127.27.17.252:33367
I20260429 20:00:37.935906 3019 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:37.936264 3021 heartbeater.cc:344] Connected to a master server at 127.27.17.253:43877
I20260429 20:00:37.936352 3021 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:37.936527 3019 heartbeater.cc:507] Master 127.27.17.252:33367 requested a full tablet report, sending...
I20260429 20:00:37.936564 3021 heartbeater.cc:507] Master 127.27.17.253:43877 requested a full tablet report, sending...
I20260429 20:00:37.937846 2887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.920256 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:42585 (local address 127.27.17.254:45045)
0429 20:00:37.920378 (+ 122us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:37.920383 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:37.921120 (+ 737us) server_negotiation.cc:408] Connection header received
0429 20:00:37.922250 (+ 1130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:37.922254 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:37.922311 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:37.922370 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:37.924280 (+ 1910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.925038 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.931866 (+ 6828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.932075 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.934325 (+ 2250us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:37.934348 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:37.934354 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:37.934388 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:37.936160 (+ 1772us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.936638 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.936641 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.936643 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.936696 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:37.936952 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:37.936955 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:37.936957 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:37.937131 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:37.937204 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:37.937554 (+ 350us) server_negotiation.cc:300] Negotiation successful
0429 20:00:37.937697 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 20:00:37.938071 2826 ts_manager.cc:194] Registered new tserver with Master: dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583)
I20260429 20:00:37.938071 2637 ts_manager.cc:194] Registered new tserver with Master: dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583)
I20260429 20:00:37.938271 3023 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:37.920539 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:42585)
0429 20:00:37.920983 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:37.921016 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:37.922032 (+ 1016us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:37.922551 (+ 519us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:37.922562 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:37.923272 (+ 710us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:37.924146 (+ 874us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.924153 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.925169 (+ 1016us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:37.925172 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:37.931728 (+ 6556us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:37.931735 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:37.932201 (+ 466us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:37.932695 (+ 494us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:37.932711 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:37.934143 (+ 1432us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:37.936311 (+ 2168us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:37.936314 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:37.936316 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:37.936515 (+ 199us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:37.936809 (+ 294us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:37.936812 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:37.936814 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:37.936860 (+ 46us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:37.937288 (+ 428us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:37.937291 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:37.937367 (+ 76us) client_negotiation.cc:770] Sending connection context
0429 20:00:37.937974 (+ 607us) client_negotiation.cc:241] Negotiation successful
0429 20:00:37.938101 (+ 127us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"spinlock_wait_cycles":8832,"thread_start_us":131,"threads_started":1}
I20260429 20:00:37.939549 3020 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45045
I20260429 20:00:37.939635 3020 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:37.939885 3020 heartbeater.cc:507] Master 127.27.17.254:45045 requested a full tablet report, sending...
I20260429 20:00:37.940725 2022 ts_manager.cc:194] Re-registered known tserver with Master: dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583)
I20260429 20:00:37.941294 2022 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:42585
W20260429 20:00:38.043754 3026 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:38.044056 3026 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:38.044164 3026 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:38.048554 3026 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:38.048657 3026 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:38.048777 3026 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:38.055706 3026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=43281
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:38.056876 3026 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:38.057854 3026 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:38.065196 3031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:38.065229 3032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:38.065234 3034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:38.065388 3026 server_base.cc:1061] running on GCE node
I20260429 20:00:38.065941 3026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:38.066563 3026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:38.067745 3026 hybrid_clock.cc:648] HybridClock initialized: now 1777492838067704 us; error 57 us; skew 500 ppm
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:38.071296 3026 init.cc:377] Logged in from keytab as oryx/127.27.17.194@KRBTEST.COM (short username oryx)
I20260429 20:00:38.072530 3026 webserver.cc:492] Webserver started at http://127.27.17.194:43281/ using document root <none> and password file <none>
I20260429 20:00:38.073208 3026 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:38.073312 3026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:38.077209 3026 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260429 20:00:38.079550 3041 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:38.080739 3026 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:00:38.080881 3026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "154fd86362f8596f8ad80fd69d8bb8ca"
server_key_iv: "dab4cb1d1021b56df9f01692ce548dcc"
server_key_version: "encryptionkey@0"
I20260429 20:00:38.081351 3026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:38.113925 3026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:38.117225 3026 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:38.117442 3026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:38.118022 3026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:38.119014 3026 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:38.119067 3026 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:38.119145 3026 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:38.119185 3026 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:38.128800 3026 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:44123
I20260429 20:00:38.128822 3154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:44123 every 8 connection(s)
I20260429 20:00:38.129853 3026 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:38.132153 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 3026
I20260429 20:00:38.132340 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 2359
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.252@KRBTEST.COM
I20260429 20:00:38.142160 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:41057
--local_ip_for_outbound_sockets=127.27.17.195
--tserver_master_addrs=127.27.17.254:45045
--webserver_port=43465
--webserver_interface=127.27.17.195
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:33239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367 with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
I20260429 20:00:38.146291 2887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.131691 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:42155 (local address 127.27.17.254:45045)
0429 20:00:38.131850 (+ 159us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.131855 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.132824 (+ 969us) server_negotiation.cc:408] Connection header received
0429 20:00:38.134196 (+ 1372us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.134200 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.134265 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.134378 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.135933 (+ 1555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.136747 (+ 814us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.137754 (+ 1007us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.137901 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.140786 (+ 2885us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.140811 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.140814 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.140845 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.143603 (+ 2758us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.144368 (+ 765us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.144375 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.144382 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.144451 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.144910 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.144914 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.144916 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.145129 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.145206 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.145975 (+ 769us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.146119 (+ 144us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260429 20:00:38.146291 2888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.131781 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:41895 (local address 127.27.17.252:33367)
0429 20:00:38.131915 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.131919 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.133607 (+ 1688us) server_negotiation.cc:408] Connection header received
0429 20:00:38.134508 (+ 901us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.134512 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.134567 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.134646 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.135985 (+ 1339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.136761 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.139165 (+ 2404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.139452 (+ 287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.141953 (+ 2501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.141978 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.141984 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.142020 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.144544 (+ 2524us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.145038 (+ 494us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.145042 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.145044 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.145091 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.145403 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.145406 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.145407 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.145559 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.145633 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.146000 (+ 367us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.146118 (+ 118us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 20:00:38.148313 3159 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.132070 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:42155)
0429 20:00:38.132665 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.132708 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.133971 (+ 1263us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.134549 (+ 578us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.134571 (+ 22us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.135417 (+ 846us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:38.135777 (+ 360us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.135789 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.136906 (+ 1117us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.136913 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.137613 (+ 700us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.137625 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.137857 (+ 232us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.138546 (+ 689us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:38.138567 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:38.140588 (+ 2021us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:38.143760 (+ 3172us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.143770 (+ 10us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.143790 (+ 20us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.144184 (+ 394us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.144620 (+ 436us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.144625 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.144627 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.144777 (+ 150us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.145290 (+ 513us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:38.145299 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:38.145631 (+ 332us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.146749 (+ 1118us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.147405 (+ 656us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":305,"thread_start_us":161,"threads_started":1}
I20260429 20:00:38.148320 3161 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.132070 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.27.17.194:41895)
0429 20:00:38.133484 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.133502 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.133998 (+ 496us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.134695 (+ 697us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.134701 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.135211 (+ 510us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:38.135810 (+ 599us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.135822 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.138189 (+ 2367us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.138195 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.139008 (+ 813us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.139022 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.139613 (+ 591us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.140138 (+ 525us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:38.140160 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:38.141790 (+ 1630us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:38.144668 (+ 2878us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.144671 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.144673 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.144914 (+ 241us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.145203 (+ 289us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.145206 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.145209 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.145271 (+ 62us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.145749 (+ 478us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:38.145754 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:38.145823 (+ 69us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.146883 (+ 1060us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.147139 (+ 256us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":27,"thread_start_us":149,"threads_started":1}
I20260429 20:00:38.149602 3156 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45045
I20260429 20:00:38.149724 3155 heartbeater.cc:344] Connected to a master server at 127.27.17.252:33367
I20260429 20:00:38.149856 3155 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:38.150466 3155 heartbeater.cc:507] Master 127.27.17.252:33367 requested a full tablet report, sending...
I20260429 20:00:38.151233 2885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.133353 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:57727 (local address 127.27.17.253:43877)
0429 20:00:38.133487 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.133490 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.139584 (+ 6094us) server_negotiation.cc:408] Connection header received
0429 20:00:38.139799 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.139803 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.140215 (+ 412us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.140278 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.141038 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.142046 (+ 1008us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.143407 (+ 1361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.143784 (+ 377us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.146256 (+ 2472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.146284 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.146291 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.146330 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.149147 (+ 2817us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.149859 (+ 712us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.149864 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.149868 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.149932 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.150252 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.150256 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.150258 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.150459 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.150571 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.150927 (+ 356us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.151069 (+ 142us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:38.151413 3160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.133195 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.27.17.194:57727)
0429 20:00:38.139465 (+ 6270us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.139480 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.139673 (+ 193us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.140415 (+ 742us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.140418 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.140632 (+ 214us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:38.140891 (+ 259us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.140902 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.142550 (+ 1648us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.142562 (+ 12us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.143250 (+ 688us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.143261 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.143933 (+ 672us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.144457 (+ 524us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:38.144478 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:38.146088 (+ 1610us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:38.149468 (+ 3380us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.149473 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.149478 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.149720 (+ 242us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.150062 (+ 342us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.150067 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.150069 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.150132 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.150585 (+ 453us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:38.150591 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:38.150693 (+ 102us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.150835 (+ 142us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.151029 (+ 194us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6179,"thread_start_us":121,"threads_started":1}
I20260429 20:00:38.151755 2826 ts_manager.cc:194] Registered new tserver with Master: b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123)
I20260429 20:00:38.152068 3157 heartbeater.cc:344] Connected to a master server at 127.27.17.253:43877
I20260429 20:00:38.152171 3157 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:38.152446 3157 heartbeater.cc:507] Master 127.27.17.253:43877 requested a full tablet report, sending...
I20260429 20:00:38.149856 3156 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:38.152884 3156 heartbeater.cc:507] Master 127.27.17.254:45045 requested a full tablet report, sending...
I20260429 20:00:38.153360 2637 ts_manager.cc:194] Registered new tserver with Master: b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123)
I20260429 20:00:38.153607 2022 ts_manager.cc:194] Re-registered known tserver with Master: b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123)
I20260429 20:00:38.154336 2022 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:42155
W20260429 20:00:38.258569 3162 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:38.258808 3162 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:38.258955 3162 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:38.262496 3162 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:38.262571 3162 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:38.262701 3162 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:38.267804 3162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:33239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:41057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=43465
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:45045,127.27.17.253:43877,127.27.17.252:33367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:38.269865 3162 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:38.271184 3162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:38.278260 3168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:38.278261 3167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:38.278340 3170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:38.278825 3162 server_base.cc:1061] running on GCE node
I20260429 20:00:38.279246 3162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:38.279817 3162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:38.281025 3162 hybrid_clock.cc:648] HybridClock initialized: now 1777492838280957 us; error 42 us; skew 500 ppm
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:38.284794 3162 init.cc:377] Logged in from keytab as oryx/127.27.17.195@KRBTEST.COM (short username oryx)
I20260429 20:00:38.286285 3162 webserver.cc:492] Webserver started at http://127.27.17.195:43465/ using document root <none> and password file <none>
I20260429 20:00:38.286974 3162 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:38.287070 3162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:38.291070 3162 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:38.293182 3177 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:38.294283 3162 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:38.294440 3162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "6e4282a3737d49509f293b9ffaf9d4cc"
format_stamp: "Formatted at 2026-04-29 20:00:35 on dist-test-slave-01nr"
server_key: "adbe742aea67d6c6a704aaa55586e980"
server_key_iv: "4cd499f55bfd327c3e1bd79ae789021a"
server_key_version: "encryptionkey@0"
I20260429 20:00:38.294862 3162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:38.328209 3162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:38.332082 3162 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:38.332333 3162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:38.332988 3162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:38.334023 3162 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:38.334074 3162 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:38.334116 3162 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:38.334174 3162 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:38.344643 3162 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:41057
I20260429 20:00:38.344727 3290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:41057 every 8 connection(s)
I20260429 20:00:38.345762 3162 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:38.350183 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 3162
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.253@KRBTEST.COM
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
Apr 29 20:00:38 dist-test-slave-01nr krb5kdc[1975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492838, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.252@KRBTEST.COM
I20260429 20:00:38.361018 2887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.349833 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:52205 (local address 127.27.17.254:45045)
0429 20:00:38.349982 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.349988 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.350004 (+ 16us) server_negotiation.cc:408] Connection header received
0429 20:00:38.350053 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.350058 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.350111 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.350191 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.351708 (+ 1517us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.352295 (+ 587us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.352935 (+ 640us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.353129 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.356778 (+ 3649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.356804 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.356811 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.356849 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.358700 (+ 1851us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.359325 (+ 625us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.359329 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.359331 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.359387 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.359758 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.359765 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.359769 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.359944 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.360024 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.360713 (+ 689us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.360843 (+ 130us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:38.361568 2888 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.347706 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:47943 (local address 127.27.17.252:33367)
0429 20:00:38.347866 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.347871 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.348636 (+ 765us) server_negotiation.cc:408] Connection header received
0429 20:00:38.349621 (+ 985us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.349626 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.349693 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.349773 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.351619 (+ 1846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.352401 (+ 782us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.354191 (+ 1790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.354416 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.357366 (+ 2950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.357394 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.357400 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.357438 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.359769 (+ 2331us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360280 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.360285 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.360287 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.360344 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360652 (+ 308us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.360659 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.360661 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.360846 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.360917 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.361282 (+ 365us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.361403 (+ 121us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 20:00:38.362504 3296 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.349001 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.253:43877 (local address 127.27.17.195:54645)
0429 20:00:38.349307 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.349321 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.349407 (+ 86us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.350108 (+ 701us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.350110 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.350746 (+ 636us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:38.351439 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.351454 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.352545 (+ 1091us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.352549 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.353214 (+ 665us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.353223 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.353366 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.354169 (+ 803us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:38.354185 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:38.356171 (+ 1986us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:38.360177 (+ 4006us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360184 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.360189 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.360398 (+ 209us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.360680 (+ 282us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360684 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.360686 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.360733 (+ 47us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.361149 (+ 416us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:38.361155 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:38.361232 (+ 77us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.361362 (+ 130us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.361633 (+ 271us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"spinlock_wait_cycles":11648,"thread_start_us":94,"threads_started":1}
I20260429 20:00:38.362514 3295 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.348021 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.252:33367 (local address 127.27.17.195:47943)
0429 20:00:38.348473 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.348508 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.349391 (+ 883us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.350070 (+ 679us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.350081 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.350746 (+ 665us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:38.351439 (+ 693us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.351455 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.352554 (+ 1099us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.352557 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.353214 (+ 657us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.353223 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.353366 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.354838 (+ 1472us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:38.354855 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:38.357196 (+ 2341us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:38.359917 (+ 2721us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.359922 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.359924 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.360147 (+ 223us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.360467 (+ 320us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360473 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.360475 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.360538 (+ 63us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.360991 (+ 453us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:38.360995 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:38.361078 (+ 83us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.361711 (+ 633us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.361825 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"spinlock_wait_cycles":14336,"thread_start_us":138,"threads_started":1}
I20260429 20:00:38.363404 2885 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.349680 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:54645 (local address 127.27.17.253:43877)
0429 20:00:38.349845 (+ 165us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.349851 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.349867 (+ 16us) server_negotiation.cc:408] Connection header received
0429 20:00:38.349910 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.349913 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.349956 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.350023 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.351619 (+ 1596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.352406 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.354517 (+ 2111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.354760 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.356913 (+ 2153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.356936 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.356940 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.356973 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.360049 (+ 3076us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360521 (+ 472us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.360524 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.360526 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.360576 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.360837 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.360840 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.360842 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.361016 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.361105 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.363061 (+ 1956us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.363181 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260429 20:00:38.363445 3297 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.349398 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:52205)
0429 20:00:38.349667 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.349680 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.349771 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.350318 (+ 547us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.350323 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.351224 (+ 901us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:38.351544 (+ 320us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.351552 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.352426 (+ 874us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.352429 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.352812 (+ 383us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.352817 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.353030 (+ 213us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.353854 (+ 824us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:38.353890 (+ 36us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:38.356621 (+ 2731us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:38.358819 (+ 2198us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.358827 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.358839 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.359190 (+ 351us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.359513 (+ 323us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:38.359516 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:38.359519 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:38.359645 (+ 126us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:38.360096 (+ 451us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:38.360101 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:38.360396 (+ 295us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.363133 (+ 2737us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.363264 (+ 131us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260429 20:00:38.364038 3291 heartbeater.cc:344] Connected to a master server at 127.27.17.252:33367
I20260429 20:00:38.364305 3291 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:38.364323 3293 heartbeater.cc:344] Connected to a master server at 127.27.17.253:43877
I20260429 20:00:38.364415 3293 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:38.364678 3292 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45045
I20260429 20:00:38.364797 3292 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:38.364969 3293 heartbeater.cc:507] Master 127.27.17.253:43877 requested a full tablet report, sending...
I20260429 20:00:38.364969 3291 heartbeater.cc:507] Master 127.27.17.252:33367 requested a full tablet report, sending...
I20260429 20:00:38.365051 3292 heartbeater.cc:507] Master 127.27.17.254:45045 requested a full tablet report, sending...
I20260429 20:00:38.366272 2637 ts_manager.cc:194] Registered new tserver with Master: 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057)
I20260429 20:00:38.366257 2022 ts_manager.cc:194] Re-registered known tserver with Master: 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057)
I20260429 20:00:38.367043 2826 ts_manager.cc:194] Registered new tserver with Master: 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057)
I20260429 20:00:38.367112 2022 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:52205
I20260429 20:00:38.368657 3305 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.355121 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.27.17.253:43877)
0429 20:00:38.355404 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.355408 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.360100 (+ 4692us) server_negotiation.cc:408] Connection header received
0429 20:00:38.360139 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.360142 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.360187 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.360263 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.361322 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.362080 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.363251 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.363433 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.365019 (+ 1586us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.365036 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.365038 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.365062 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.367022 (+ 1960us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.367485 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.367490 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.367492 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.367539 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.367815 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.367818 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.367820 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.367976 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.368099 (+ 123us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.368419 (+ 320us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.368534 (+ 115us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260429 20:00:38.373867 3302 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.354154 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.27.17.252:33367)
0429 20:00:38.356078 (+ 1924us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.356081 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.356090 (+ 9us) server_negotiation.cc:408] Connection header received
0429 20:00:38.356118 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.356120 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.356159 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.356260 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.357494 (+ 1234us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.358301 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.359832 (+ 1531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.360017 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.370079 (+ 10062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.370099 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.370102 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.370138 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.372271 (+ 2133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.372739 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.372743 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.372747 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.372797 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.373026 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.373030 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.373032 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.373187 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.373401 (+ 214us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.373545 (+ 144us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.373665 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1859,"thread_start_us":86,"threads_started":1}
I20260429 20:00:38.375245 2887 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.355061 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.27.17.254:45045)
0429 20:00:38.361499 (+ 6438us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.361503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.361514 (+ 11us) server_negotiation.cc:408] Connection header received
0429 20:00:38.361544 (+ 30us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.361547 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.361587 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.361931 (+ 344us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.369199 (+ 7268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.370023 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.370815 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.371555 (+ 740us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.372084 (+ 529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.372102 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.372105 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.372129 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.373744 (+ 1615us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.374155 (+ 411us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.374159 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.374160 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.374204 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.374411 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.374414 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.374415 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.374572 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.374674 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.375033 (+ 359us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.375129 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6373,"thread_start_us":79,"threads_started":1}
I20260429 20:00:38.378108 2022 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:38.381069 2022 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:38.398113 3316 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.393594 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:34583 (local address 127.0.0.1:56486)
0429 20:00:38.393882 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.393897 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.393993 (+ 96us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.395052 (+ 1059us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.395055 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.395073 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:38.395355 (+ 282us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.395364 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.396926 (+ 1562us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.396929 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.397749 (+ 820us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.397756 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.397836 (+ 80us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.397851 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.397895 (+ 44us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.397954 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
I20260429 20:00:38.399219 3320 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.395178 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:41057 (local address 127.0.0.1:56380)
0429 20:00:38.395541 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.395561 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.395888 (+ 327us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.396458 (+ 570us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.396461 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.396481 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:38.396705 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.396710 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.397805 (+ 1095us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.397809 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.398755 (+ 946us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.398764 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.398916 (+ 152us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.398977 (+ 61us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.399035 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.399091 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":108,"threads_started":1}
I20260429 20:00:38.399282 3319 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.394256 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.27.17.193:34583)
0429 20:00:38.394546 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.394575 (+ 29us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.394592 (+ 17us) server_negotiation.cc:408] Connection header received
0429 20:00:38.394696 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.394700 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.394867 (+ 167us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.395685 (+ 818us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:38.395818 (+ 133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.396816 (+ 998us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.398374 (+ 1558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.398875 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.398978 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.399045 (+ 67us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.399124 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260429 20:00:38.399281 3318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.394087 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37232 (local address 127.27.17.194:44123)
0429 20:00:38.394431 (+ 344us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.394438 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.394460 (+ 22us) server_negotiation.cc:408] Connection header received
0429 20:00:38.394561 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.394566 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.394770 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.394955 (+ 185us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:38.395529 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.396624 (+ 1095us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.398045 (+ 1421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.398810 (+ 765us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.398978 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.399045 (+ 67us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.399129 (+ 84us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":122,"threads_started":1}
I20260429 20:00:38.399595 3317 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.393938 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:44123 (local address 127.0.0.1:37232)
0429 20:00:38.394209 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.394225 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.394325 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.395125 (+ 800us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.395129 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.395150 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:38.395395 (+ 245us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.395403 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.396759 (+ 1356us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.396763 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.397721 (+ 958us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.397731 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.397835 (+ 104us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.397851 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.397904 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.397954 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":106,"threads_started":1}
I20260429 20:00:38.400413 3321 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.395700 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.27.17.195:41057)
0429 20:00:38.396107 (+ 407us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.396111 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.396127 (+ 16us) server_negotiation.cc:408] Connection header received
0429 20:00:38.396197 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.396201 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.396322 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.396471 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:38.396820 (+ 349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.397545 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.398927 (+ 1382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.400022 (+ 1095us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.400120 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.400201 (+ 81us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.400280 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":105,"threads_started":1}
I20260429 20:00:38.402148 3089 tablet_service.cc:1511] Processing CreateTablet for tablet 6c666f20f42a40499dbd5e489e9c8a35 (DEFAULT_TABLE table=test-table [id=41130c6feada49d08b3fb3fbc7ae0ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:38.402624 2953 tablet_service.cc:1511] Processing CreateTablet for tablet 6c666f20f42a40499dbd5e489e9c8a35 (DEFAULT_TABLE table=test-table [id=41130c6feada49d08b3fb3fbc7ae0ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:38.403333 3089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c666f20f42a40499dbd5e489e9c8a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:38.403565 2953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c666f20f42a40499dbd5e489e9c8a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:38.403294 3225 tablet_service.cc:1511] Processing CreateTablet for tablet 6c666f20f42a40499dbd5e489e9c8a35 (DEFAULT_TABLE table=test-table [id=41130c6feada49d08b3fb3fbc7ae0ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:38.404121 3225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c666f20f42a40499dbd5e489e9c8a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:38.409814 3322 tablet_bootstrap.cc:492] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Bootstrap starting.
I20260429 20:00:38.411264 3323 tablet_bootstrap.cc:492] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Bootstrap starting.
I20260429 20:00:38.412117 3324 tablet_bootstrap.cc:492] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Bootstrap starting.
I20260429 20:00:38.412420 3322 tablet_bootstrap.cc:654] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:38.413322 3322 log.cc:826] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:38.413805 3323 tablet_bootstrap.cc:654] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:38.414069 3324 tablet_bootstrap.cc:654] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:38.414636 3323 log.cc:826] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:38.414814 3324 log.cc:826] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:38.415565 3322 tablet_bootstrap.cc:492] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: No bootstrap required, opened a new log
I20260429 20:00:38.415795 3322 ts_tablet_manager.cc:1403] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260429 20:00:38.416599 3323 tablet_bootstrap.cc:492] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: No bootstrap required, opened a new log
I20260429 20:00:38.416793 3323 ts_tablet_manager.cc:1403] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 20:00:38.419121 3322 raft_consensus.cc:359] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.419373 3322 raft_consensus.cc:385] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:38.419432 3322 raft_consensus.cc:740] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02cb1f1f1d14632b3eb9eb718d6ff1e, State: Initialized, Role: FOLLOWER
I20260429 20:00:38.419889 3322 consensus_queue.cc:260] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.419889 3323 raft_consensus.cc:359] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.420073 3323 raft_consensus.cc:385] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:38.420130 3323 raft_consensus.cc:740] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc242720f061411e98271e1a9ad950b1, State: Initialized, Role: FOLLOWER
I20260429 20:00:38.420539 3324 tablet_bootstrap.cc:492] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: No bootstrap required, opened a new log
I20260429 20:00:38.420542 3323 consensus_queue.cc:260] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.420771 3324 ts_tablet_manager.cc:1403] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260429 20:00:38.420872 3322 ts_tablet_manager.cc:1434] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 20:00:38.421360 3323 ts_tablet_manager.cc:1434] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 20:00:38.421562 3020 heartbeater.cc:499] Master 127.27.17.254:45045 was elected leader, sending a full tablet report...
I20260429 20:00:38.423805 3156 heartbeater.cc:499] Master 127.27.17.254:45045 was elected leader, sending a full tablet report...
I20260429 20:00:38.424132 3324 raft_consensus.cc:359] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.424361 3324 raft_consensus.cc:385] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:38.424419 3324 raft_consensus.cc:740] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4282a3737d49509f293b9ffaf9d4cc, State: Initialized, Role: FOLLOWER
I20260429 20:00:38.424858 3324 consensus_queue.cc:260] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.425773 3324 ts_tablet_manager.cc:1434] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260429 20:00:38.426123 3292 heartbeater.cc:499] Master 127.27.17.254:45045 was elected leader, sending a full tablet report...
I20260429 20:00:38.451481 3328 raft_consensus.cc:493] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:38.451683 3328 raft_consensus.cc:515] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.452725 3328 leader_election.cc:290] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583), 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057)
I20260429 20:00:38.456054 3160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.453044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:34583 (local address 127.27.17.194:41463)
0429 20:00:38.453186 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.453203 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.453300 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.453631 (+ 331us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.453634 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.453655 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:38.453890 (+ 235us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.453897 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.454830 (+ 933us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.454833 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.455588 (+ 755us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.455596 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.455732 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.455750 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.455805 (+ 55us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.455861 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260429 20:00:38.456068 3161 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.453040 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:41057 (local address 127.27.17.194:58781)
0429 20:00:38.453334 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:38.453348 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:38.453453 (+ 105us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:38.453738 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:38.453741 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:38.453759 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:38.453933 (+ 174us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.453960 (+ 27us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.454794 (+ 834us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.454798 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:38.455585 (+ 787us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:38.455593 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.455740 (+ 147us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.455756 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 20:00:38.455809 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 20:00:38.455860 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":93}
I20260429 20:00:38.456535 3321 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.453358 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:58781 (local address 127.27.17.195:41057)
0429 20:00:38.453493 (+ 135us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.453497 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.453508 (+ 11us) server_negotiation.cc:408] Connection header received
0429 20:00:38.453550 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.453552 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.453622 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.453701 (+ 79us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:38.454080 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.454583 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.455719 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.456222 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.456266 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.456336 (+ 70us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.456400 (+ 64us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 20:00:38.456600 3319 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.453223 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:41463 (local address 127.27.17.193:34583)
0429 20:00:38.453383 (+ 160us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.453388 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.453405 (+ 17us) server_negotiation.cc:408] Connection header received
0429 20:00:38.453459 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.453463 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.453520 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.453597 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:38.454021 (+ 424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.454582 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.455742 (+ 1160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.456275 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.456317 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.456373 (+ 56us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.456440 (+ 67us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260429 20:00:38.457188 2973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c666f20f42a40499dbd5e489e9c8a35" candidate_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc242720f061411e98271e1a9ad950b1" is_pre_election: true
I20260429 20:00:38.457196 3245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c666f20f42a40499dbd5e489e9c8a35" candidate_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" is_pre_election: true
I20260429 20:00:38.457577 3245 raft_consensus.cc:2468] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b02cb1f1f1d14632b3eb9eb718d6ff1e in term 0.
I20260429 20:00:38.457577 2973 raft_consensus.cc:2468] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b02cb1f1f1d14632b3eb9eb718d6ff1e in term 0.
I20260429 20:00:38.458173 3044 leader_election.cc:304] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4282a3737d49509f293b9ffaf9d4cc, b02cb1f1f1d14632b3eb9eb718d6ff1e; no voters:
I20260429 20:00:38.458504 3328 raft_consensus.cc:2804] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:38.458729 3328 raft_consensus.cc:493] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:38.458806 3328 raft_consensus.cc:3060] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:38.459874 3328 raft_consensus.cc:515] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.460274 3328 leader_election.cc:290] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [CANDIDATE]: Term 1 election: Requested vote from peers dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583), 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057)
I20260429 20:00:38.460743 2973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c666f20f42a40499dbd5e489e9c8a35" candidate_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc242720f061411e98271e1a9ad950b1"
I20260429 20:00:38.460753 3245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c666f20f42a40499dbd5e489e9c8a35" candidate_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4282a3737d49509f293b9ffaf9d4cc"
I20260429 20:00:38.460861 2973 raft_consensus.cc:3060] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:38.460894 3245 raft_consensus.cc:3060] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:38.461885 3245 raft_consensus.cc:2468] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b02cb1f1f1d14632b3eb9eb718d6ff1e in term 1.
I20260429 20:00:38.461898 2973 raft_consensus.cc:2468] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b02cb1f1f1d14632b3eb9eb718d6ff1e in term 1.
I20260429 20:00:38.462311 3042 leader_election.cc:304] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b02cb1f1f1d14632b3eb9eb718d6ff1e, dc242720f061411e98271e1a9ad950b1; no voters:
I20260429 20:00:38.462522 3328 raft_consensus.cc:2804] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:38.462769 3328 raft_consensus.cc:697] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 LEADER]: Becoming Leader. State: Replica: b02cb1f1f1d14632b3eb9eb718d6ff1e, State: Running, Role: LEADER
I20260429 20:00:38.463384 3328 consensus_queue.cc:237] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } }
I20260429 20:00:38.467496 2022 catalog_manager.cc:5671] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e reported cstate change: term changed from 0 to 1, leader changed from <none> to b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194). New cstate: current_term: 1 leader_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:38.482244 3318 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.479231 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.27.17.194:44123)
0429 20:00:38.479412 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.479416 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.479433 (+ 17us) server_negotiation.cc:408] Connection header received
0429 20:00:38.479528 (+ 95us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.479531 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.479583 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.479688 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:38.480049 (+ 361us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.480553 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.481207 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.481406 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.481486 (+ 80us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:38.481841 (+ 355us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:38.481963 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.482041 (+ 78us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.482089 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260429 20:00:38.488701 3245 raft_consensus.cc:1275] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Refusing update from remote peer b02cb1f1f1d14632b3eb9eb718d6ff1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:38.488701 2973 raft_consensus.cc:1275] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 1 FOLLOWER]: Refusing update from remote peer b02cb1f1f1d14632b3eb9eb718d6ff1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:38.489497 3331 consensus_queue.cc:1048] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:38.489814 3328 consensus_queue.cc:1048] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:38.499620 3336 mvcc.cc:204] Tried to move back new op lower bound from 7280610666441875456 to 7280610666348871680. Current Snapshot: MvccSnapshot[applied={T|T < 7280610666441875456}]
I20260429 20:00:38.509580 2023 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:38.509776 2023 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:38.514508 2023 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4a284ad344d34d8c9e4cf992f4bb42d4: Sending DeleteTablet for 3 replicas of tablet 6c666f20f42a40499dbd5e489e9c8a35
I20260429 20:00:38.515837 3225 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c666f20f42a40499dbd5e489e9c8a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:38 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:56380
I20260429 20:00:38.515918 2953 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c666f20f42a40499dbd5e489e9c8a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:38 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:56486
I20260429 20:00:38.516462 3347 tablet_replica.cc:333] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: stopping tablet replica
I20260429 20:00:38.516424 3089 tablet_service.cc:1558] Processing DeleteTablet for tablet 6c666f20f42a40499dbd5e489e9c8a35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:38 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:37232
I20260429 20:00:38.516779 3347 raft_consensus.cc:2243] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:38.516955 3348 tablet_replica.cc:333] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: stopping tablet replica
I20260429 20:00:38.517145 3347 raft_consensus.cc:2272] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:38.517308 3348 raft_consensus.cc:2243] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:00:38.517930 3346 tablet_replica.cc:333] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: stopping tablet replica
I20260429 20:00:38.518172 3346 raft_consensus.cc:2243] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:38.518482 3346 raft_consensus.cc:2272] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:38.519233 3347 ts_tablet_manager.cc:1916] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:38.519928 3346 ts_tablet_manager.cc:1916] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:38.520749 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 1991
I20260429 20:00:38.521680 3348 raft_consensus.cc:2272] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:38.523628 3346 ts_tablet_manager.cc:1929] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:38.523728 3346 log.cc:1199] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/6c666f20f42a40499dbd5e489e9c8a35
I20260429 20:00:38.524065 3346 ts_tablet_manager.cc:1950] T 6c666f20f42a40499dbd5e489e9c8a35 P dc242720f061411e98271e1a9ad950b1: Deleting consensus metadata
I20260429 20:00:38.525523 3348 ts_tablet_manager.cc:1916] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:38.525548 3347 ts_tablet_manager.cc:1929] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:38.525776 3347 log.cc:1199] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/6c666f20f42a40499dbd5e489e9c8a35
I20260429 20:00:38.526190 3347 ts_tablet_manager.cc:1950] T 6c666f20f42a40499dbd5e489e9c8a35 P 6e4282a3737d49509f293b9ffaf9d4cc: Deleting consensus metadata
W20260429 20:00:38.528350 3178 connection.cc:570] server connection from 127.0.0.1:56380 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 20:00:38.528245 3348 ts_tablet_manager.cc:1929] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260429 20:00:38.529433 2908 connection.cc:463] server connection from 127.0.0.1:56486 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56486 (request call id 1) could send its response
I20260429 20:00:38.529506 3348 log.cc:1199] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/6c666f20f42a40499dbd5e489e9c8a35
I20260429 20:00:38.529850 3348 ts_tablet_manager.cc:1950] T 6c666f20f42a40499dbd5e489e9c8a35 P b02cb1f1f1d14632b3eb9eb718d6ff1e: Deleting consensus metadata
W20260429 20:00:38.530711 3043 connection.cc:463] server connection from 127.0.0.1:37232 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37232 (request call id 1) could send its response
I20260429 20:00:38.540938 3302 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.531806 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.27.17.252:33367)
0429 20:00:38.531932 (+ 126us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.531936 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.532230 (+ 294us) server_negotiation.cc:408] Connection header received
0429 20:00:38.532463 (+ 233us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.532467 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.532531 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.532591 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.533761 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.534552 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.535497 (+ 945us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.535696 (+ 199us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.537335 (+ 1639us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.537356 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.537360 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.537393 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.539464 (+ 2071us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.539915 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.539918 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.539920 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.539968 (+ 48us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.540216 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.540219 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.540220 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.540352 (+ 132us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.540443 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.540700 (+ 257us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.540800 (+ 100us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260429 20:00:38.541931 3305 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:38.532158 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.27.17.253:43877)
0429 20:00:38.532288 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:38.532293 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:38.532490 (+ 197us) server_negotiation.cc:408] Connection header received
0429 20:00:38.532685 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:38.532688 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:38.532748 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:38.532844 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:38.534000 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.534628 (+ 628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:38.535452 (+ 824us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:38.535676 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:38.537144 (+ 1468us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:38.537172 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:38.537174 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:38.537200 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:38.539082 (+ 1882us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.539521 (+ 439us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.539525 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.539527 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.539579 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:38.539821 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:38.539824 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:38.539826 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:38.540042 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:38.540163 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:38.541229 (+ 1066us) server_negotiation.cc:300] Negotiation successful
0429 20:00:38.541780 (+ 551us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260429 20:00:39.503530 3355 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:39.502844 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:42517)
0429 20:00:39.503177 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:39.503388 (+ 211us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":139,"threads_started":1}
W20260429 20:00:39.503885 3020 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:45045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:39.508134 3356 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:39.507290 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:34443)
0429 20:00:39.507951 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:39.508022 (+ 71us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":76,"threads_started":1}
I20260429 20:00:39.508385 3357 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:39.507933 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:46305)
0429 20:00:39.508187 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:39.508263 (+ 76us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
W20260429 20:00:39.508395 3292 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:45045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
W20260429 20:00:39.508689 3156 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:45045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:40.015614 3358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a284ad344d34d8c9e4cf992f4bb42d4)
I20260429 20:00:40.015825 3359 raft_consensus.cc:493] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a284ad344d34d8c9e4cf992f4bb42d4)
I20260429 20:00:40.015851 3358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:40.015949 3359 raft_consensus.cc:515] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:40.016482 3359 leader_election.cc:290] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253:43877)
I20260429 20:00:40.017005 3358 leader_election.cc:290] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252:33367)
I20260429 20:00:40.017261 2647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49609ff57952474f8fdb3b87bf50a50a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" is_pre_election: true
I20260429 20:00:40.017333 3360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.016750 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38028)
0429 20:00:40.017037 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.017212 (+ 175us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260429 20:00:40.017402 2647 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49609ff57952474f8fdb3b87bf50a50a in term 1.
I20260429 20:00:40.017565 2829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49609ff57952474f8fdb3b87bf50a50a" is_pre_election: true
I20260429 20:00:40.017623 3361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.017130 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38034)
0429 20:00:40.017472 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.017531 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":55,"threads_started":1}
W20260429 20:00:40.017694 2813 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
W20260429 20:00:40.017763 2624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:40.017774 2829 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee5c1d0c7ac4d118e868a8aa6a57ce3 in term 1.
I20260429 20:00:40.018030 2813 leader_election.cc:304] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49609ff57952474f8fdb3b87bf50a50a, 4ee5c1d0c7ac4d118e868a8aa6a57ce3; no voters:
I20260429 20:00:40.018196 2622 leader_election.cc:304] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49609ff57952474f8fdb3b87bf50a50a, 4ee5c1d0c7ac4d118e868a8aa6a57ce3; no voters:
I20260429 20:00:40.018246 3359 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260429 20:00:40.018302 3359 raft_consensus.cc:493] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a284ad344d34d8c9e4cf992f4bb42d4)
I20260429 20:00:40.018343 3359 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 1 FOLLOWER]: Advancing to term 2
I20260429 20:00:40.018420 3358 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260429 20:00:40.018476 3358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a284ad344d34d8c9e4cf992f4bb42d4)
I20260429 20:00:40.018517 3358 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 1 FOLLOWER]: Advancing to term 2
I20260429 20:00:40.018577 3360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.018291 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38050)
0429 20:00:40.018442 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.018493 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260429 20:00:40.018699 2813 leader_election.cc:336] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:40.019134 3361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.018834 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38052)
0429 20:00:40.019033 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.019074 (+ 41us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
W20260429 20:00:40.019224 2624 leader_election.cc:336] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:40.019644 3359 raft_consensus.cc:515] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:40.019730 3358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:40.020066 3359 leader_election.cc:290] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 2 election: Requested vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 4ee5c1d0c7ac4d118e868a8aa6a57ce3 (127.27.17.253:43877)
I20260429 20:00:40.020162 3358 leader_election.cc:290] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 2 election: Requested vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252:33367)
I20260429 20:00:40.020476 2829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49609ff57952474f8fdb3b87bf50a50a"
I20260429 20:00:40.020581 3361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.020230 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38072)
0429 20:00:40.020469 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.020516 (+ 47us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260429 20:00:40.020594 3360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.020161 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38058)
0429 20:00:40.020460 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.020506 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260429 20:00:40.020658 2829 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ee5c1d0c7ac4d118e868a8aa6a57ce3 in current term 2: Already voted for candidate 49609ff57952474f8fdb3b87bf50a50a in this term.
I20260429 20:00:40.020589 2647 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49609ff57952474f8fdb3b87bf50a50a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3"
I20260429 20:00:40.020819 2647 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 49609ff57952474f8fdb3b87bf50a50a in current term 2: Already voted for candidate 4ee5c1d0c7ac4d118e868a8aa6a57ce3 in this term.
I20260429 20:00:40.021600 3361 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.021383 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38092)
0429 20:00:40.021491 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.021537 (+ 46us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260429 20:00:40.021672 3360 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.021226 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38084)
0429 20:00:40.021560 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.021602 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254}
W20260429 20:00:40.021729 2624 leader_election.cc:336] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
W20260429 20:00:40.021775 2813 leader_election.cc:336] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:40.021806 2624 leader_election.cc:304] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ee5c1d0c7ac4d118e868a8aa6a57ce3; no voters: 49609ff57952474f8fdb3b87bf50a50a, 4a284ad344d34d8c9e4cf992f4bb42d4
I20260429 20:00:40.021831 2813 leader_election.cc:304] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 49609ff57952474f8fdb3b87bf50a50a; no voters: 4a284ad344d34d8c9e4cf992f4bb42d4, 4ee5c1d0c7ac4d118e868a8aa6a57ce3
I20260429 20:00:40.021956 3359 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260429 20:00:40.022039 3358 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260429 20:00:40.505239 3364 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.504652 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:37677)
0429 20:00:40.505047 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.505115 (+ 68us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":91,"threads_started":1}
I20260429 20:00:40.509380 3365 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.508932 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:53993)
0429 20:00:40.509220 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.509280 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260429 20:00:40.509680 3366 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:40.509265 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:51873)
0429 20:00:40.509521 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:40.509574 (+ 53us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260429 20:00:41.507005 3367 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.506327 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:32785)
0429 20:00:41.506794 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.506859 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":114,"threads_started":1}
I20260429 20:00:41.510820 3368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.510277 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:46013)
0429 20:00:41.510635 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.510699 (+ 64us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":100,"threads_started":1}
I20260429 20:00:41.510952 3369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.510398 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:42191)
0429 20:00:41.510726 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.510785 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":93,"threads_started":1}
I20260429 20:00:41.774919 3370 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:41.775074 3370 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:41.775646 3370 leader_election.cc:290] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252:33367)
I20260429 20:00:41.776194 2829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49609ff57952474f8fdb3b87bf50a50a" is_pre_election: true
I20260429 20:00:41.776501 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.775947 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38356)
0429 20:00:41.776253 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.776328 (+ 75us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260429 20:00:41.776365 2829 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ee5c1d0c7ac4d118e868a8aa6a57ce3 in term 2.
I20260429 20:00:41.776901 2622 leader_election.cc:304] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49609ff57952474f8fdb3b87bf50a50a, 4ee5c1d0c7ac4d118e868a8aa6a57ce3; no voters:
I20260429 20:00:41.777125 3370 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260429 20:00:41.777215 3370 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:41.777273 3370 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 2 FOLLOWER]: Advancing to term 3
I20260429 20:00:41.777808 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.777396 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38360)
0429 20:00:41.777654 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.777727 (+ 73us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260429 20:00:41.777937 2624 leader_election.cc:336] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:41.778654 3370 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:41.779104 3370 leader_election.cc:290] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 3 election: Requested vote from peers 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045), 49609ff57952474f8fdb3b87bf50a50a (127.27.17.252:33367)
I20260429 20:00:41.779438 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.779207 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38362)
0429 20:00:41.779341 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.779380 (+ 39us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260429 20:00:41.779739 2829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "49609ff57952474f8fdb3b87bf50a50a"
I20260429 20:00:41.779965 2829 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 2 FOLLOWER]: Advancing to term 3
I20260429 20:00:41.780335 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.780082 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38370)
0429 20:00:41.780185 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.780277 (+ 92us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260429 20:00:41.780455 2624 leader_election.cc:336] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
I20260429 20:00:41.781167 2829 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ee5c1d0c7ac4d118e868a8aa6a57ce3 in term 3.
I20260429 20:00:41.781525 2622 leader_election.cc:304] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 49609ff57952474f8fdb3b87bf50a50a, 4ee5c1d0c7ac4d118e868a8aa6a57ce3; no voters: 4a284ad344d34d8c9e4cf992f4bb42d4
I20260429 20:00:41.781754 3370 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 3 FOLLOWER]: Leader election won for term 3
I20260429 20:00:41.781955 3370 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [term 3 LEADER]: Becoming Leader. State: Replica: 4ee5c1d0c7ac4d118e868a8aa6a57ce3, State: Running, Role: LEADER
I20260429 20:00:41.782213 3370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } }
I20260429 20:00:41.783473 3373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee5c1d0c7ac4d118e868a8aa6a57ce3. Latest consensus state: current_term: 3 leader_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:41.783628 3373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:41.783912 3375 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:41.786362 3375 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:41.786823 3375 catalog_manager.cc:1269] Loaded cluster ID: dae3886cdad348fc95dcb8ee81b09b07
I20260429 20:00:41.786873 3375 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:41.787508 3375 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:41.787916 3375 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Loaded TSK: 0
I20260429 20:00:41.788544 3375 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:00:41.843634 2636 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:41.846066 2636 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:41.849009 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.848767 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38388)
0429 20:00:41.848897 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.848942 (+ 45us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260429 20:00:41.849056 2829 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [term 3 FOLLOWER]: Refusing update from remote peer 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260429 20:00:41.849579 3370 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260429 20:00:41.849888 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.849628 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.0.0.1:38394)
0429 20:00:41.849778 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.849829 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260429 20:00:41.850010 2624 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 -> Peer 4a284ad344d34d8c9e4cf992f4bb42d4 (127.27.17.254:45045): Couldn't send request to peer 4a284ad344d34d8c9e4cf992f4bb42d4. Status: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260429 20:00:41.851110 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee5c1d0c7ac4d118e868a8aa6a57ce3. Latest consensus state: current_term: 3 leader_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:41.851246 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:41.853000 3370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:41.853330 3377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4a284ad344d34d8c9e4cf992f4bb42d4" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45045 } } peers { permanent_uuid: "4ee5c1d0c7ac4d118e868a8aa6a57ce3" member_type: VOTER last_known_addr { host: "127.27.17.253" port: 43877 } attrs { promote: false } } peers { permanent_uuid: "49609ff57952474f8fdb3b87bf50a50a" member_type: VOTER last_known_addr { host: "127.27.17.252" port: 33367 } attrs { promote: false } } }
I20260429 20:00:41.853516 3377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49609ff57952474f8fdb3b87bf50a50a [sys.catalog]: This master's current role is: FOLLOWER
I20260429 20:00:41.853991 3370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:41.864686 3371 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.860824 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:41057 (local address 127.0.0.1:56392)
0429 20:00:41.860998 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.861015 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:41.861107 (+ 92us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:41.861573 (+ 466us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:41.861576 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:41.861596 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:41.861866 (+ 270us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:41.861874 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.863237 (+ 1363us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:41.863241 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:41.864168 (+ 927us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:41.864175 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.864247 (+ 72us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:41.864263 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 20:00:41.864300 (+ 37us) client_negotiation.cc:241] Negotiation successful
0429 20:00:41.864460 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260429 20:00:41.865257 3384 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.861340 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:34583 (local address 127.0.0.1:56500)
0429 20:00:41.861879 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.861893 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:41.861977 (+ 84us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:41.862579 (+ 602us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:41.862583 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:41.862604 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:41.862929 (+ 325us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:41.862940 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.863837 (+ 897us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:41.863843 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:41.864553 (+ 710us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:41.864562 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.864715 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:41.864732 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 20:00:41.865071 (+ 339us) client_negotiation.cc:241] Negotiation successful
0429 20:00:41.865131 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":167,"threads_started":1}
I20260429 20:00:41.865662 3385 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.861611 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.27.17.193:34583)
0429 20:00:41.862068 (+ 457us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:41.862074 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:41.862191 (+ 117us) server_negotiation.cc:408] Connection header received
0429 20:00:41.862248 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:41.862251 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:41.862340 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:41.862469 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:41.863068 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.863721 (+ 653us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:41.864702 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.865359 (+ 657us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:41.865402 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:41.865455 (+ 53us) server_negotiation.cc:300] Negotiation successful
0429 20:00:41.865511 (+ 56us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":81,"threads_started":1}
I20260429 20:00:41.865748 3383 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.860929 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56392 (local address 127.27.17.195:41057)
0429 20:00:41.861158 (+ 229us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:41.861162 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:41.861175 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:41.861219 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:41.861221 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:41.861261 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:41.861995 (+ 734us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:41.862110 (+ 115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.863117 (+ 1007us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:41.864968 (+ 1851us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.865463 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:41.865511 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:41.865562 (+ 51us) server_negotiation.cc:300] Negotiation successful
0429 20:00:41.865619 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260429 20:00:41.866291 3225 tablet_service.cc:1511] Processing CreateTablet for tablet 7fe8bdea72744ffea38d78ea4538395c (DEFAULT_TABLE table=test-table [id=19b12d718a4b436499d54dae156aa41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:41.866293 2953 tablet_service.cc:1511] Processing CreateTablet for tablet 7fe8bdea72744ffea38d78ea4538395c (DEFAULT_TABLE table=test-table [id=19b12d718a4b436499d54dae156aa41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:41.866532 3387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.861878 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.27.17.194:44123)
0429 20:00:41.862157 (+ 279us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:41.862161 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:41.862417 (+ 256us) server_negotiation.cc:408] Connection header received
0429 20:00:41.862479 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:41.862482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:41.862577 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:41.862669 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:41.863142 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.863944 (+ 802us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:41.865495 (+ 1551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.866235 (+ 740us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:41.866270 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:41.866328 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 20:00:41.866388 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260429 20:00:41.866626 2953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe8bdea72744ffea38d78ea4538395c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:41.865396 3386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.861756 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:44123 (local address 127.0.0.1:37244)
0429 20:00:41.862291 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.862316 (+ 25us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:41.862404 (+ 88us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:41.862698 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:41.862701 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:41.862719 (+ 18us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:41.863000 (+ 281us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:41.863007 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.864082 (+ 1075us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:41.864086 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:41.865038 (+ 952us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:41.865046 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:41.865145 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:41.865160 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 20:00:41.865207 (+ 47us) client_negotiation.cc:241] Negotiation successful
0429 20:00:41.865276 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":84,"threads_started":1}
I20260429 20:00:41.866641 3225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe8bdea72744ffea38d78ea4538395c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:41.867226 3089 tablet_service.cc:1511] Processing CreateTablet for tablet 7fe8bdea72744ffea38d78ea4538395c (DEFAULT_TABLE table=test-table [id=19b12d718a4b436499d54dae156aa41f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:41.867506 3089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fe8bdea72744ffea38d78ea4538395c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:41.868809 3388 tablet_bootstrap.cc:492] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1: Bootstrap starting.
I20260429 20:00:41.869158 3389 tablet_bootstrap.cc:492] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Bootstrap starting.
I20260429 20:00:41.869905 3388 tablet_bootstrap.cc:654] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:41.870517 3389 tablet_bootstrap.cc:654] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:41.871011 3390 tablet_bootstrap.cc:492] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Bootstrap starting.
I20260429 20:00:41.872085 3390 tablet_bootstrap.cc:654] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:41.872643 3388 tablet_bootstrap.cc:492] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1: No bootstrap required, opened a new log
I20260429 20:00:41.872771 3388 ts_tablet_manager.cc:1403] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260429 20:00:41.873237 3390 tablet_bootstrap.cc:492] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: No bootstrap required, opened a new log
I20260429 20:00:41.873356 3390 ts_tablet_manager.cc:1403] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260429 20:00:41.873457 3388 raft_consensus.cc:359] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:41.873581 3388 raft_consensus.cc:385] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:41.873613 3388 raft_consensus.cc:740] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc242720f061411e98271e1a9ad950b1, State: Initialized, Role: FOLLOWER
I20260429 20:00:41.873790 3388 consensus_queue.cc:260] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:41.873785 3390 raft_consensus.cc:359] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:41.873898 3390 raft_consensus.cc:385] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:41.873937 3390 raft_consensus.cc:740] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02cb1f1f1d14632b3eb9eb718d6ff1e, State: Initialized, Role: FOLLOWER
I20260429 20:00:41.874074 3390 consensus_queue.cc:260] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:41.874339 3388 ts_tablet_manager.cc:1434] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:41.874447 3390 ts_tablet_manager.cc:1434] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 20:00:41.875160 3367 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.874796 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:36167)
0429 20:00:41.875009 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.875069 (+ 60us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260429 20:00:41.875459 3369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.875192 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:36641)
0429 20:00:41.875319 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.875369 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260429 20:00:41.876593 3389 tablet_bootstrap.cc:492] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: No bootstrap required, opened a new log
I20260429 20:00:41.876719 3389 ts_tablet_manager.cc:1403] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260429 20:00:41.877259 3389 raft_consensus.cc:359] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:41.877447 3389 raft_consensus.cc:385] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:41.877475 3021 heartbeater.cc:499] Master 127.27.17.253:43877 was elected leader, sending a full tablet report...
I20260429 20:00:41.877808 3367 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.877539 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:54809)
0429 20:00:41.877669 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.877728 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260429 20:00:41.877532 3389 raft_consensus.cc:740] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4282a3737d49509f293b9ffaf9d4cc, State: Initialized, Role: FOLLOWER
I20260429 20:00:41.878064 3389 consensus_queue.cc:260] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:41.878480 3389 ts_tablet_manager.cc:1434] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:41.878641 3369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.878370 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:60319)
0429 20:00:41.878515 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.878571 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260429 20:00:41.879413 3157 heartbeater.cc:499] Master 127.27.17.253:43877 was elected leader, sending a full tablet report...
I20260429 20:00:41.880416 3368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.880094 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:35185)
0429 20:00:41.880289 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.880340 (+ 51us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260429 20:00:41.881356 3293 heartbeater.cc:499] Master 127.27.17.253:43877 was elected leader, sending a full tablet report...
I20260429 20:00:41.881932 3368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:41.881645 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:33583)
0429 20:00:41.881785 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:41.881846 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260429 20:00:41.883890 3158 tablet.cc:2404] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 20:00:41.923112 3022 tablet.cc:2404] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 20:00:42.100468 3294 tablet.cc:2404] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 20:00:42.167732 3394 raft_consensus.cc:493] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:42.167886 3394 raft_consensus.cc:515] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:42.168931 3394 leader_election.cc:290] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057), b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123)
I20260429 20:00:42.172273 3367 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.169279 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:41057 (local address 127.27.17.193:44067)
0429 20:00:42.169462 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.169479 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:42.169614 (+ 135us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:42.169918 (+ 304us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:42.169924 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:42.169954 (+ 30us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:42.170232 (+ 278us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:42.170242 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.171205 (+ 963us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:42.171209 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:42.171974 (+ 765us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:42.171981 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.172070 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:42.172084 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 20:00:42.172120 (+ 36us) client_negotiation.cc:241] Negotiation successful
0429 20:00:42.172169 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260429 20:00:42.172915 3383 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.169445 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:44067 (local address 127.27.17.195:41057)
0429 20:00:42.169630 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:42.169635 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:42.169680 (+ 45us) server_negotiation.cc:408] Connection header received
0429 20:00:42.169739 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:42.169742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:42.169795 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:42.169881 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:42.170392 (+ 511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.171093 (+ 701us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:42.172111 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.172644 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:42.172684 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:42.172738 (+ 54us) server_negotiation.cc:300] Negotiation successful
0429 20:00:42.172797 (+ 59us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260429 20:00:42.172931 3397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.169581 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:44123 (local address 127.27.17.193:35259)
0429 20:00:42.169935 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.169947 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:42.170038 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:42.170377 (+ 339us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:42.170381 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:42.170402 (+ 21us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:42.170699 (+ 297us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:42.170705 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.171734 (+ 1029us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:42.171737 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:42.172549 (+ 812us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:42.172560 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.172688 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:42.172705 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 20:00:42.172756 (+ 51us) client_negotiation.cc:241] Negotiation successful
0429 20:00:42.172805 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260429 20:00:42.173347 3245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fe8bdea72744ffea38d78ea4538395c" candidate_uuid: "dc242720f061411e98271e1a9ad950b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" is_pre_election: true
I20260429 20:00:42.173388 3387 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.169752 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:35259 (local address 127.27.17.194:44123)
0429 20:00:42.169903 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:42.169907 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:42.170032 (+ 125us) server_negotiation.cc:408] Connection header received
0429 20:00:42.170114 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:42.170117 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:42.170161 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:42.170306 (+ 145us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:42.170848 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.171592 (+ 744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:42.172700 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.173116 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:42.173151 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:42.173201 (+ 50us) server_negotiation.cc:300] Negotiation successful
0429 20:00:42.173254 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:42.173509 3245 raft_consensus.cc:2468] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc242720f061411e98271e1a9ad950b1 in term 0.
I20260429 20:00:42.173903 2908 leader_election.cc:304] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4282a3737d49509f293b9ffaf9d4cc, dc242720f061411e98271e1a9ad950b1; no voters:
I20260429 20:00:42.174029 3109 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fe8bdea72744ffea38d78ea4538395c" candidate_uuid: "dc242720f061411e98271e1a9ad950b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" is_pre_election: true
I20260429 20:00:42.174175 3394 raft_consensus.cc:2804] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:42.174273 3394 raft_consensus.cc:493] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:42.174264 3109 raft_consensus.cc:2468] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc242720f061411e98271e1a9ad950b1 in term 0.
I20260429 20:00:42.174332 3394 raft_consensus.cc:3060] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:42.175392 3394 raft_consensus.cc:515] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:42.175753 3394 leader_election.cc:290] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [CANDIDATE]: Term 1 election: Requested vote from peers 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057), b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123)
I20260429 20:00:42.176192 3245 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fe8bdea72744ffea38d78ea4538395c" candidate_uuid: "dc242720f061411e98271e1a9ad950b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4282a3737d49509f293b9ffaf9d4cc"
I20260429 20:00:42.176330 3109 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fe8bdea72744ffea38d78ea4538395c" candidate_uuid: "dc242720f061411e98271e1a9ad950b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e"
I20260429 20:00:42.176412 3245 raft_consensus.cc:3060] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:42.176437 3109 raft_consensus.cc:3060] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:42.177532 3109 raft_consensus.cc:2468] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc242720f061411e98271e1a9ad950b1 in term 1.
I20260429 20:00:42.177532 3245 raft_consensus.cc:2468] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc242720f061411e98271e1a9ad950b1 in term 1.
I20260429 20:00:42.177868 2908 leader_election.cc:304] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e4282a3737d49509f293b9ffaf9d4cc, dc242720f061411e98271e1a9ad950b1; no voters:
I20260429 20:00:42.178099 3394 raft_consensus.cc:2804] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:42.178390 3394 raft_consensus.cc:697] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [term 1 LEADER]: Becoming Leader. State: Replica: dc242720f061411e98271e1a9ad950b1, State: Running, Role: LEADER
I20260429 20:00:42.178661 3394 consensus_queue.cc:237] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } }
I20260429 20:00:42.179536 3397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.179234 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:35627)
0429 20:00:42.179395 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.179445 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260429 20:00:42.181233 2637 catalog_manager.cc:5671] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc242720f061411e98271e1a9ad950b1 (127.27.17.193). New cstate: current_term: 1 leader_uuid: "dc242720f061411e98271e1a9ad950b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc242720f061411e98271e1a9ad950b1" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 34583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:42.237912 3385 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.234578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.27.17.193:34583)
0429 20:00:42.234752 (+ 174us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:42.234755 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:42.234769 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:42.235041 (+ 272us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:42.235045 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:42.235111 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:42.235209 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:42.235584 (+ 375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.236100 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:42.236782 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:42.236952 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:42.237007 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:42.237435 (+ 428us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:42.237529 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:42.237657 (+ 128us) server_negotiation.cc:300] Negotiation successful
0429 20:00:42.237706 (+ 49us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260429 20:00:42.240978 3245 raft_consensus.cc:1275] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Refusing update from remote peer dc242720f061411e98271e1a9ad950b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:42.241118 3109 raft_consensus.cc:1275] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Refusing update from remote peer dc242720f061411e98271e1a9ad950b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:42.241592 3394 consensus_queue.cc:1048] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4282a3737d49509f293b9ffaf9d4cc" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 41057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:42.241765 3398 consensus_queue.cc:1048] T 7fe8bdea72744ffea38d78ea4538395c P dc242720f061411e98271e1a9ad950b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02cb1f1f1d14632b3eb9eb718d6ff1e" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:42.243647 3368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.243305 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:60659)
0429 20:00:42.243504 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.243562 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260429 20:00:42.243965 3369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.243740 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:40741)
0429 20:00:42.243843 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.243893 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260429 20:00:42.247321 3369 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.246969 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.194:52355)
0429 20:00:42.247185 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.247235 (+ 50us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260429 20:00:42.250190 3397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.247516 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:41813)
0429 20:00:42.250031 (+ 2515us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.250096 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2410}
I20260429 20:00:42.251343 3403 mvcc.cc:204] Tried to move back new op lower bound from 7280610681813524480 to 7280610681567010816. Current Snapshot: MvccSnapshot[applied={T|T < 7280610681813524480 or (T in {7280610681813524480})}]
I20260429 20:00:42.251569 3397 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.251262 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.193:41279)
0429 20:00:42.251413 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.251461 (+ 48us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260429 20:00:42.256247 3368 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.255917 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45045 (local address 127.27.17.195:58303)
0429 20:00:42.256089 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.256156 (+ 67us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:45045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260429 20:00:42.258620 2636 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:42.258793 2636 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:42.262965 2636 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 4ee5c1d0c7ac4d118e868a8aa6a57ce3: Sending DeleteTablet for 3 replicas of tablet 7fe8bdea72744ffea38d78ea4538395c
I20260429 20:00:42.264024 3225 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fe8bdea72744ffea38d78ea4538395c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:42 UTC) from {username='oryx', principal='oryx/127.27.17.253@KRBTEST.COM'} at 127.0.0.1:56392
I20260429 20:00:42.264451 3411 tablet_replica.cc:333] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: stopping tablet replica
I20260429 20:00:42.264678 3411 raft_consensus.cc:2243] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:42.264727 3089 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fe8bdea72744ffea38d78ea4538395c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:42 UTC) from {username='oryx', principal='oryx/127.27.17.253@KRBTEST.COM'} at 127.0.0.1:37244
I20260429 20:00:42.264918 3411 raft_consensus.cc:2272] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:42.265051 2953 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fe8bdea72744ffea38d78ea4538395c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:42 UTC) from {username='oryx', principal='oryx/127.27.17.253@KRBTEST.COM'} at 127.0.0.1:56500
I20260429 20:00:42.265539 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 2890
I20260429 20:00:42.267313 3412 tablet_replica.cc:333] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: stopping tablet replica
I20260429 20:00:42.267436 3411 ts_tablet_manager.cc:1916] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:42.267468 3412 raft_consensus.cc:2243] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:42.267820 3412 raft_consensus.cc:2272] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:42.268573 3412 ts_tablet_manager.cc:1916] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:42.270829 3411 ts_tablet_manager.cc:1929] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:42.270958 3411 log.cc:1199] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/7fe8bdea72744ffea38d78ea4538395c
I20260429 20:00:42.271068 3412 ts_tablet_manager.cc:1929] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:42.271178 3412 log.cc:1199] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/7fe8bdea72744ffea38d78ea4538395c
I20260429 20:00:42.271317 3411 ts_tablet_manager.cc:1950] T 7fe8bdea72744ffea38d78ea4538395c P 6e4282a3737d49509f293b9ffaf9d4cc: Deleting consensus metadata
I20260429 20:00:42.271606 3412 ts_tablet_manager.cc:1950] T 7fe8bdea72744ffea38d78ea4538395c P b02cb1f1f1d14632b3eb9eb718d6ff1e: Deleting consensus metadata
I20260429 20:00:42.272315 2623 catalog_manager.cc:5002] TS b02cb1f1f1d14632b3eb9eb718d6ff1e (127.27.17.194:44123): tablet 7fe8bdea72744ffea38d78ea4538395c (table test-table [id=19b12d718a4b436499d54dae156aa41f]) successfully deleted
I20260429 20:00:42.272544 2623 catalog_manager.cc:5002] TS 6e4282a3737d49509f293b9ffaf9d4cc (127.27.17.195:41057): tablet 7fe8bdea72744ffea38d78ea4538395c (table test-table [id=19b12d718a4b436499d54dae156aa41f]) successfully deleted
I20260429 20:00:42.274075 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 3026
I20260429 20:00:42.274529 3386 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:42.274014 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:34583 (local address 127.0.0.1:56524)
0429 20:00:42.274367 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:42.274428 (+ 61us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.193:34583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260429 20:00:42.275907 2621 catalog_manager.cc:4729] TS dc242720f061411e98271e1a9ad950b1 (127.27.17.193:34583): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fe8bdea72744ffea38d78ea4538395c: Network error: Client connection negotiation failed: client connection to 127.27.17.193:34583: connect: Connection refused (error 111)
I20260429 20:00:42.281554 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 3162
I20260429 20:00:42.288798 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 2495
I20260429 20:00:42.296074 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 2617
2026-04-29T20:00:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9658 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:42 dist-test-slave-01nr krb5kdc[3416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:42 dist-test-slave-01nr krb5kdc[3416](info): set up 2 sockets
Apr 29 20:00:42 dist-test-slave-01nr krb5kdc[3416](info): commencing operation
krb5kdc: starting...
W20260429 20:00:44.356802 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:44 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:44.518797 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:36801
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:36801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:44.623930 3432 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:44.624243 3432 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:44.624364 3432 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:44.627817 3432 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:44.627925 3432 flags.cc:432] Enabled experimental flag: --txn_manager_enabled=true
W20260429 20:00:44.627960 3432 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:44.628010 3432 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:44.628054 3432 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:44.628095 3432 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:44.632663 3432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:36801
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:36801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:44.633985 3432 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:44.634933 3432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:44.640964 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:44.640978 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:44.641028 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:44.641113 3432 server_base.cc:1061] running on GCE node
I20260429 20:00:44.641848 3432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:44.642740 3432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:44.643993 3432 hybrid_clock.cc:648] HybridClock initialized: now 1777492844643955 us; error 39 us; skew 500 ppm
Apr 29 20:00:44 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492844, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:44.647269 3432 init.cc:377] Logged in from keytab as oryx/127.27.17.254@KRBTEST.COM (short username oryx)
I20260429 20:00:44.648424 3432 webserver.cc:492] Webserver started at http://127.27.17.254:41097/ using document root <none> and password file <none>
I20260429 20:00:44.649012 3432 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:44.649091 3432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:44.649307 3432 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:44.651449 3432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc"
format_stamp: "Formatted at 2026-04-29 20:00:44 on dist-test-slave-01nr"
server_key: "94a6845a07de5a06378421927ebac9eb"
server_key_iv: "08fab76de9853d6ecd41e22be2c85425"
server_key_version: "encryptionkey@0"
I20260429 20:00:44.651948 3432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc"
format_stamp: "Formatted at 2026-04-29 20:00:44 on dist-test-slave-01nr"
server_key: "94a6845a07de5a06378421927ebac9eb"
server_key_iv: "08fab76de9853d6ecd41e22be2c85425"
server_key_version: "encryptionkey@0"
I20260429 20:00:44.655534 3432 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:00:44.657856 3447 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:44.659078 3432 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:44.659235 3432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc"
format_stamp: "Formatted at 2026-04-29 20:00:44 on dist-test-slave-01nr"
server_key: "94a6845a07de5a06378421927ebac9eb"
server_key_iv: "08fab76de9853d6ecd41e22be2c85425"
server_key_version: "encryptionkey@0"
I20260429 20:00:44.659394 3432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:44.683782 3432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:44.687237 3432 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:44.687485 3432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:44.698525 3432 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:36801
I20260429 20:00:44.698786 3509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:36801 every 8 connection(s)
I20260429 20:00:44.699936 3432 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:44.702886 3510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:44.705868 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 3432
I20260429 20:00:44.705983 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:44.706255 27719 external_mini_cluster.cc:1468] Setting key be8cae702df4702c1dae0bb85490e3c1
I20260429 20:00:44.709252 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Bootstrap starting.
I20260429 20:00:44.711984 3510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:44.712775 3510 log.cc:826] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Log is configured to *not* fsync() on all Append() calls
Apr 29 20:00:44 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:44.715384 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: No bootstrap required, opened a new log
I20260429 20:00:44.719077 3510 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36801 } }
I20260429 20:00:44.719321 3510 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:44.719416 3510 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f8efd6d15fb45aaa8a4a38ab53aeecc, State: Initialized, Role: FOLLOWER
I20260429 20:00:44.719923 3510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36801 } }
I20260429 20:00:44.720052 3510 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:44.720134 3510 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:44.720235 3510 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:44.720935 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:44.708004 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48686 (local address 127.27.17.254:36801)
0429 20:00:44.708694 (+ 690us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:44.708704 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:44.708743 (+ 39us) server_negotiation.cc:408] Connection header received
0429 20:00:44.709310 (+ 567us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:44.709326 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:44.709704 (+ 378us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:44.709989 (+ 285us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:44.711202 (+ 1213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:44.712009 (+ 807us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:44.712766 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:44.713075 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:44.715713 (+ 2638us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:44.715735 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:44.715747 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:44.715776 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:44.717733 (+ 1957us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:44.718310 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:44.718315 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:44.718320 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:44.718411 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:44.718730 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:44.718733 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:44.718735 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:44.719162 (+ 427us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:44.719388 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:44.719747 (+ 359us) server_negotiation.cc:300] Negotiation successful
0429 20:00:44.720172 (+ 425us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":158,"threads_started":1}
I20260429 20:00:44.721158 3510 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36801 } }
I20260429 20:00:44.721535 3510 leader_election.cc:304] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f8efd6d15fb45aaa8a4a38ab53aeecc; no voters:
I20260429 20:00:44.721837 3510 leader_election.cc:290] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:44.721984 3515 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:44.722204 3515 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [term 1 LEADER]: Becoming Leader. State: Replica: 2f8efd6d15fb45aaa8a4a38ab53aeecc, State: Running, Role: LEADER
I20260429 20:00:44.722564 3515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36801 } }
I20260429 20:00:44.723129 3510 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:44.723855 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f8efd6d15fb45aaa8a4a38ab53aeecc. Latest consensus state: current_term: 1 leader_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36801 } } }
I20260429 20:00:44.724094 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:44.724242 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f8efd6d15fb45aaa8a4a38ab53aeecc" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 36801 } } }
I20260429 20:00:44.724328 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:44.724643 3522 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:44.727675 3522 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:44.734769 3522 catalog_manager.cc:1357] Generated new cluster ID: 9c334052365941f283808e4d825f8d18
I20260429 20:00:44.734884 3522 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:44.771677 3522 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:44.772636 3522 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:44.781724 3522 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Generated new TSK 0
I20260429 20:00:44.782500 3522 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:44.851817 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:36801
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:44.963743 3538 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:44.963973 3538 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:44.964035 3538 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:44.967494 3538 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:44.967571 3538 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:44.967635 3538 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 20:00:44.967679 3538 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:44.972096 3538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:36801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:44.973313 3538 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:44.974152 3538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:44.981099 3544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:44.981137 3543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:44.981099 3546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:44.981455 3538 server_base.cc:1061] running on GCE node
I20260429 20:00:44.981880 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:44.982563 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:44.983757 3538 hybrid_clock.cc:648] HybridClock initialized: now 1777492844983733 us; error 40 us; skew 500 ppm
Apr 29 20:00:44 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492844, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:44.986598 3538 init.cc:377] Logged in from keytab as oryx/127.27.17.193@KRBTEST.COM (short username oryx)
I20260429 20:00:44.987797 3538 webserver.cc:492] Webserver started at http://127.27.17.193:38579/ using document root <none> and password file <none>
I20260429 20:00:44.988373 3538 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:44.988449 3538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:44.988667 3538 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:44.990463 3538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "214cb678006b45e585f008d4627b5cb0"
format_stamp: "Formatted at 2026-04-29 20:00:44 on dist-test-slave-01nr"
server_key: "1e292f33eb51e6f7fbaefab6e77b91bf"
server_key_iv: "9d55fb7f818bcb9dce00de6acd246c80"
server_key_version: "encryptionkey@0"
I20260429 20:00:44.991041 3538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "214cb678006b45e585f008d4627b5cb0"
format_stamp: "Formatted at 2026-04-29 20:00:44 on dist-test-slave-01nr"
server_key: "1e292f33eb51e6f7fbaefab6e77b91bf"
server_key_iv: "9d55fb7f818bcb9dce00de6acd246c80"
server_key_version: "encryptionkey@0"
I20260429 20:00:44.994558 3538 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:00:44.997016 3553 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:44.998272 3538 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:44.998382 3538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "214cb678006b45e585f008d4627b5cb0"
format_stamp: "Formatted at 2026-04-29 20:00:44 on dist-test-slave-01nr"
server_key: "1e292f33eb51e6f7fbaefab6e77b91bf"
server_key_iv: "9d55fb7f818bcb9dce00de6acd246c80"
server_key_version: "encryptionkey@0"
I20260429 20:00:44.998543 3538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:45.036517 3538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:45.039637 3538 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:45.039835 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:45.041298 3538 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:45.041378 3538 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:45.041450 3538 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:45.041501 3538 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492844, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:45.052908 3538 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:33871
I20260429 20:00:45.052945 3668 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:33871 every 8 connection(s)
I20260429 20:00:45.053905 3538 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:45.055269 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.042081 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:59311 (local address 127.27.17.254:36801)
0429 20:00:45.042245 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.042250 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.042941 (+ 691us) server_negotiation.cc:408] Connection header received
0429 20:00:45.043863 (+ 922us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.043867 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.043932 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.044036 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.045679 (+ 1643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.046574 (+ 895us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.047576 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.047820 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.050192 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.050225 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.050232 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.050270 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.052275 (+ 2005us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.053029 (+ 754us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.053034 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.053038 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.053145 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.053558 (+ 413us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.053565 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.053571 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.053742 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.053867 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.055034 (+ 1167us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.055151 (+ 117us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:45.055346 3562 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.042338 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.27.17.193:59311)
0429 20:00:45.042765 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.042803 (+ 38us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.043648 (+ 845us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.044233 (+ 585us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.044243 (+ 10us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.044835 (+ 592us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:45.045516 (+ 681us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.045530 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.046691 (+ 1161us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.046696 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.047401 (+ 705us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.047411 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.047664 (+ 253us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.048327 (+ 663us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:45.048351 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:45.050017 (+ 1666us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:45.052441 (+ 2424us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.052447 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.052459 (+ 12us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.052884 (+ 425us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.053305 (+ 421us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.053310 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.053312 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.053420 (+ 108us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.053867 (+ 447us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:45.053874 (+ 7us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:45.054145 (+ 271us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.054377 (+ 232us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.054639 (+ 262us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":137,"threads_started":1}
I20260429 20:00:45.056514 3669 heartbeater.cc:344] Connected to a master server at 127.27.17.254:36801
I20260429 20:00:45.056696 3669 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:45.057200 3669 heartbeater.cc:507] Master 127.27.17.254:36801 requested a full tablet report, sending...
I20260429 20:00:45.058708 3464 ts_manager.cc:194] Registered new tserver with Master: 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871)
I20260429 20:00:45.058871 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 3538
I20260429 20:00:45.059013 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:00:45.059298 27719 external_mini_cluster.cc:1468] Setting key 34030519c17bccddd184d09ccd51bb95
I20260429 20:00:45.060322 3464 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:59311
I20260429 20:00:45.066870 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.059448 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:36557 (local address 127.27.17.254:36801)
0429 20:00:45.059556 (+ 108us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.059560 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.059574 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:45.059853 (+ 279us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.059856 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.059902 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.059974 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.060710 (+ 736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.061470 (+ 760us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.062410 (+ 940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.062657 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.063507 (+ 850us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.063527 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.063529 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.063591 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.065152 (+ 1561us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.065680 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.065683 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.065684 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.065744 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.066056 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.066059 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.066061 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.066210 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.066314 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.066628 (+ 314us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.066736 (+ 108us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260429 20:00:45.067019 3675 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.059110 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.27.17.193:36557)
0429 20:00:45.059454 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.059471 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.059579 (+ 108us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.060067 (+ 488us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.060070 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.060300 (+ 230us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:45.060579 (+ 279us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.060587 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.061613 (+ 1026us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.061620 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.062222 (+ 602us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.062234 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.062415 (+ 181us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.062992 (+ 577us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:45.063006 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:45.063345 (+ 339us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:45.065282 (+ 1937us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.065286 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.065289 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.065555 (+ 266us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.065864 (+ 309us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.065870 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.065873 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.065948 (+ 75us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.066322 (+ 374us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:45.066327 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:45.066441 (+ 114us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.066570 (+ 129us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.066760 (+ 190us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for oryx/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:45.119145 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:36801
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:45.224275 3680 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:45.224630 3680 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:45.224745 3680 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:45.229372 3680 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:45.229449 3680 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:45.229506 3680 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 20:00:45.229619 3680 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:45.234161 3680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:36801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:45.235461 3680 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:45.236284 3680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:45.243134 3685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:45.243156 3686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:45.243134 3688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:45.243659 3680 server_base.cc:1061] running on GCE node
I20260429 20:00:45.244036 3680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:45.244621 3680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:45.245828 3680 hybrid_clock.cc:648] HybridClock initialized: now 1777492845245801 us; error 38 us; skew 500 ppm
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492845, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:45.248749 3680 init.cc:377] Logged in from keytab as oryx/127.27.17.194@KRBTEST.COM (short username oryx)
I20260429 20:00:45.249974 3680 webserver.cc:492] Webserver started at http://127.27.17.194:43525/ using document root <none> and password file <none>
I20260429 20:00:45.250602 3680 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:45.250685 3680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:45.250962 3680 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:45.252775 3680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "0117ac2a4f834ed18442f237d154d200"
format_stamp: "Formatted at 2026-04-29 20:00:45 on dist-test-slave-01nr"
server_key: "f0832c6e1701c0dccab5c76cb2548609"
server_key_iv: "ea39ec2e2d259285fd566403c716ee73"
server_key_version: "encryptionkey@0"
I20260429 20:00:45.253340 3680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "0117ac2a4f834ed18442f237d154d200"
format_stamp: "Formatted at 2026-04-29 20:00:45 on dist-test-slave-01nr"
server_key: "f0832c6e1701c0dccab5c76cb2548609"
server_key_iv: "ea39ec2e2d259285fd566403c716ee73"
server_key_version: "encryptionkey@0"
I20260429 20:00:45.257191 3680 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260429 20:00:45.260180 3695 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:45.261513 3680 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260429 20:00:45.261693 3680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "0117ac2a4f834ed18442f237d154d200"
format_stamp: "Formatted at 2026-04-29 20:00:45 on dist-test-slave-01nr"
server_key: "f0832c6e1701c0dccab5c76cb2548609"
server_key_iv: "ea39ec2e2d259285fd566403c716ee73"
server_key_version: "encryptionkey@0"
I20260429 20:00:45.261824 3680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:45.283166 3680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:45.286783 3680 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:45.287066 3680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:45.288642 3680 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:45.288697 3680 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:45.288738 3680 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:45.288800 3680 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492845, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:45.300843 3680 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:38141
I20260429 20:00:45.300905 3810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:38141 every 8 connection(s)
I20260429 20:00:45.302268 3680 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:45.302846 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.289508 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:33791 (local address 127.27.17.254:36801)
0429 20:00:45.289650 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.289653 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.290535 (+ 882us) server_negotiation.cc:408] Connection header received
0429 20:00:45.291580 (+ 1045us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.291584 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.291652 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.291757 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.293490 (+ 1733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.294381 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.295383 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.295654 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.298250 (+ 2596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.298280 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.298283 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.298344 (+ 61us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.300277 (+ 1933us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.301022 (+ 745us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.301026 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.301030 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.301104 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.301562 (+ 458us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.301566 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.301567 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.301749 (+ 182us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.301845 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.302488 (+ 643us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.302631 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 20:00:45.303699 3704 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.289822 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.27.17.194:33791)
0429 20:00:45.290356 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.290389 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.291319 (+ 930us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.291925 (+ 606us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.291937 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.292570 (+ 633us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:45.293271 (+ 701us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.293291 (+ 20us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.294611 (+ 1320us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.294617 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.295213 (+ 596us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.295223 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.295541 (+ 318us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.296270 (+ 729us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:45.296298 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:45.298048 (+ 1750us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:45.300473 (+ 2425us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.300480 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.300494 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.300863 (+ 369us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.301239 (+ 376us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.301243 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.301246 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.301367 (+ 121us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.301872 (+ 505us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:45.301880 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:45.302155 (+ 275us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.302401 (+ 246us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.302704 (+ 303us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260429 20:00:45.305007 3811 heartbeater.cc:344] Connected to a master server at 127.27.17.254:36801
I20260429 20:00:45.305202 3811 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:45.305799 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 3680
I20260429 20:00:45.305912 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:45.305891 3811 heartbeater.cc:507] Master 127.27.17.254:36801 requested a full tablet report, sending...
I20260429 20:00:45.306264 27719 external_mini_cluster.cc:1468] Setting key daa906443d2beaf6e09fed46987eac23
I20260429 20:00:45.307152 3464 ts_manager.cc:194] Registered new tserver with Master: 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141)
I20260429 20:00:45.307899 3464 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:33791
WARNING: no policy specified for oryx/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.195@KRBTEST.COM" created.
I20260429 20:00:45.319661 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.307556 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:40737 (local address 127.27.17.254:36801)
0429 20:00:45.307674 (+ 118us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.307678 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.308073 (+ 395us) server_negotiation.cc:408] Connection header received
0429 20:00:45.308232 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.308236 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.308285 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.308373 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.308925 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.309455 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.310245 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.310398 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.315774 (+ 5376us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.315810 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.315817 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.315860 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.317569 (+ 1709us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.318131 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.318135 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.318136 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.318197 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.318585 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.318591 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.318594 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.318866 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.319040 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.319410 (+ 370us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.319532 (+ 122us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 20:00:45.319797 3817 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.307575 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.27.17.194:40737)
0429 20:00:45.307995 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.308011 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.308126 (+ 115us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.308391 (+ 265us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.308394 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.308596 (+ 202us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:45.308803 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.308809 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.309572 (+ 763us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.309578 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.310113 (+ 535us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.310124 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.310257 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.315125 (+ 4868us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:45.315145 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:45.315506 (+ 361us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:45.317771 (+ 2265us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.317776 (+ 5us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.317779 (+ 3us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.317989 (+ 210us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.318371 (+ 382us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.318378 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.318383 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.318453 (+ 70us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.319037 (+ 584us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:45.319041 (+ 4us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:45.319160 (+ 119us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.319302 (+ 142us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.319471 (+ 169us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab.
I20260429 20:00:45.364107 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:36801
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:45.473405 3822 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:45.473691 3822 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:45.473783 3822 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:45.477208 3822 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:45.477313 3822 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:45.477406 3822 flags.cc:432] Enabled experimental flag: --enable_txn_system_client_init=true
W20260429 20:00:45.477504 3822 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:45.482692 3822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/krb5kdc/oryx.keytab
--principal=oryx/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.17.254:36801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:45.483907 3822 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:45.484777 3822 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:45.491637 3830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:45.491631 3828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:45.491631 3827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:45.492076 3822 server_base.cc:1061] running on GCE node
I20260429 20:00:45.492498 3822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:45.493127 3822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:45.494323 3822 hybrid_clock.cc:648] HybridClock initialized: now 1777492845494292 us; error 50 us; skew 500 ppm
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492845, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:45.497098 3822 init.cc:377] Logged in from keytab as oryx/127.27.17.195@KRBTEST.COM (short username oryx)
I20260429 20:00:45.498198 3822 webserver.cc:492] Webserver started at http://127.27.17.195:39103/ using document root <none> and password file <none>
I20260429 20:00:45.498716 3822 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:45.498762 3822 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:45.498986 3822 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:45.500747 3822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "706bab00e1d34a7592c131fb3fca0e59"
format_stamp: "Formatted at 2026-04-29 20:00:45 on dist-test-slave-01nr"
server_key: "5ab71986a26b7f2723dd87aa593d3617"
server_key_iv: "09c1a44e418f0786a80cf6f0c78a34d1"
server_key_version: "encryptionkey@0"
I20260429 20:00:45.501187 3822 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "706bab00e1d34a7592c131fb3fca0e59"
format_stamp: "Formatted at 2026-04-29 20:00:45 on dist-test-slave-01nr"
server_key: "5ab71986a26b7f2723dd87aa593d3617"
server_key_iv: "09c1a44e418f0786a80cf6f0c78a34d1"
server_key_version: "encryptionkey@0"
I20260429 20:00:45.504742 3822 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260429 20:00:45.507014 3837 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:45.508041 3822 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:45.508147 3822 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "706bab00e1d34a7592c131fb3fca0e59"
format_stamp: "Formatted at 2026-04-29 20:00:45 on dist-test-slave-01nr"
server_key: "5ab71986a26b7f2723dd87aa593d3617"
server_key_iv: "09c1a44e418f0786a80cf6f0c78a34d1"
server_key_version: "encryptionkey@0"
I20260429 20:00:45.508232 3822 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:45.542264 3822 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:45.545300 3822 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:45.545467 3822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:45.546859 3822 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:45.546973 3822 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:45.547073 3822 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:45.547129 3822 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492845, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:45.559212 3822 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:42599
I20260429 20:00:45.560425 3822 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:45.560900 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 3822
I20260429 20:00:45.560943 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.547863 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:39367 (local address 127.27.17.254:36801)
0429 20:00:45.547999 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.548002 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.548817 (+ 815us) server_negotiation.cc:408] Connection header received
0429 20:00:45.549798 (+ 981us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.549803 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.549877 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.549976 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.551508 (+ 1532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.552348 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.553214 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.553417 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.556431 (+ 3014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.556458 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.556461 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.556503 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.558504 (+ 2001us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.559206 (+ 702us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.559245 (+ 39us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.559246 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.559304 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.559694 (+ 390us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.559697 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.559698 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.559871 (+ 173us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.559961 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.560602 (+ 641us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.560762 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260429 20:00:45.561012 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:45.561302 27719 external_mini_cluster.cc:1468] Setting key 709d33ac8841550d09f7ad8073171c3d
I20260429 20:00:45.560998 3952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:42599 every 8 connection(s)
I20260429 20:00:45.561546 3846 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.548185 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.27.17.195:39367)
0429 20:00:45.548665 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.548698 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.549517 (+ 819us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.550151 (+ 634us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.550165 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.550777 (+ 612us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:45.551363 (+ 586us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.551375 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.552509 (+ 1134us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.552514 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.553010 (+ 496us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.553019 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.553293 (+ 274us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.554094 (+ 801us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:45.554136 (+ 42us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:45.556238 (+ 2102us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:45.558662 (+ 2424us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.558668 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.558679 (+ 11us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.559072 (+ 393us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.559441 (+ 369us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.559445 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.559447 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.559573 (+ 126us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.559983 (+ 410us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:45.559988 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:45.560250 (+ 262us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.560466 (+ 216us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.560744 (+ 278us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":154,"threads_started":1}
I20260429 20:00:45.563526 3953 heartbeater.cc:344] Connected to a master server at 127.27.17.254:36801
I20260429 20:00:45.563719 3953 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:45.564216 3953 heartbeater.cc:507] Master 127.27.17.254:36801 requested a full tablet report, sending...
I20260429 20:00:45.565301 3464 ts_manager.cc:194] Registered new tserver with Master: 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:45.566088 3464 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:39367
I20260429 20:00:45.573185 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.565918 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:37555 (local address 127.27.17.254:36801)
0429 20:00:45.566050 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.566053 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.566486 (+ 433us) server_negotiation.cc:408] Connection header received
0429 20:00:45.566566 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.566569 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.566620 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.566843 (+ 223us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.567696 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.568172 (+ 476us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.568804 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.569004 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.569931 (+ 927us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.569949 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.569953 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.569985 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.571647 (+ 1662us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.572071 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.572075 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.572077 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.572130 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.572390 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.572395 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.572397 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.572552 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.572661 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.572907 (+ 246us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.573009 (+ 102us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260429 20:00:45.573256 3959 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.565809 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.27.17.195:37555)
0429 20:00:45.566301 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.566320 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.566476 (+ 156us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.567047 (+ 571us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.567051 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.567270 (+ 219us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:45.567570 (+ 300us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.567578 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.568285 (+ 707us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.568291 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.568686 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.568693 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.568820 (+ 127us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.569456 (+ 636us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:45.569469 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:45.569776 (+ 307us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:45.571777 (+ 2001us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.571781 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.571783 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.571967 (+ 184us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.572248 (+ 281us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:45.572251 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:45.572252 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:45.572301 (+ 49us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:45.572651 (+ 350us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:45.572654 (+ 3us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:45.572747 (+ 93us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.572856 (+ 109us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.573001 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260429 20:00:45.575556 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492845, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 20:00:45.598533 3513 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:45.592535 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45648 (local address 127.27.17.254:36801)
0429 20:00:45.592682 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.592686 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.592700 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:45.592742 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.592745 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.592790 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.592866 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.593709 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.594234 (+ 525us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.595008 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.595164 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.598365 (+ 3201us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45648: BlockingRecv error: recv got EOF from 127.0.0.1:45648 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Apr 29 20:00:45 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492845, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:45.608928 3513 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.600104 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.27.17.254:36801)
0429 20:00:45.600294 (+ 190us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.600300 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.600345 (+ 45us) server_negotiation.cc:408] Connection header received
0429 20:00:45.600531 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.600535 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.600581 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.600668 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:45.601394 (+ 726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.601907 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.602576 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.602791 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.605547 (+ 2756us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:45.605572 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:45.605575 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:45.605611 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:45.607435 (+ 1824us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.607869 (+ 434us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.607872 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.607873 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.607926 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:45.608142 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:45.608144 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:45.608145 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:45.608292 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:45.608424 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.608704 (+ 280us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.608803 (+ 99us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 20:00:45.612079 3464 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:45.614645 3464 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:45.627838 3979 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.623602 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:42599 (local address 127.0.0.1:44802)
0429 20:00:45.624371 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.624386 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.624471 (+ 85us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.624929 (+ 458us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.624932 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.624952 (+ 20us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:45.625210 (+ 258us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.625218 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.626388 (+ 1170us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.626393 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.627318 (+ 925us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.627326 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.627455 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.627520 (+ 65us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.627615 (+ 95us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.627685 (+ 70us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":130,"threads_started":1}
I20260429 20:00:45.628129 3977 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.623968 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:33871 (local address 127.0.0.1:40578)
0429 20:00:45.624763 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.624776 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.624912 (+ 136us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.625444 (+ 532us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.625447 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.625475 (+ 28us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:45.625714 (+ 239us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.625719 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.626977 (+ 1258us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.626981 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.627787 (+ 806us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.627794 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.627887 (+ 93us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.627901 (+ 14us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.627949 (+ 48us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.628000 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":84,"threads_started":1}
I20260429 20:00:45.628412 3978 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.623809 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.27.17.195:42599)
0429 20:00:45.624171 (+ 362us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.624176 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.624456 (+ 280us) server_negotiation.cc:408] Connection header received
0429 20:00:45.624627 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.624635 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.624794 (+ 159us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.624923 (+ 129us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:45.625334 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.626274 (+ 940us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.627456 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.628002 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.628109 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.628170 (+ 61us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.628276 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":109,"threads_started":1}
I20260429 20:00:45.628803 3976 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.623152 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.27.17.194:38141)
0429 20:00:45.623478 (+ 326us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.623487 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.623979 (+ 492us) server_negotiation.cc:408] Connection header received
0429 20:00:45.624066 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.624071 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.624249 (+ 178us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.624444 (+ 195us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:45.625177 (+ 733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.626351 (+ 1174us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.627728 (+ 1377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.628265 (+ 537us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.628359 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.628582 (+ 223us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.628661 (+ 79us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":147,"threads_started":1}
I20260429 20:00:45.629142 3975 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.622981 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:38141 (local address 127.0.0.1:57988)
0429 20:00:45.623652 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.623689 (+ 37us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.623843 (+ 154us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.624619 (+ 776us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.624624 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.624647 (+ 23us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:45.624975 (+ 328us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.624985 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.626635 (+ 1650us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.626639 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.627574 (+ 935us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.627583 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.628419 (+ 836us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.628431 (+ 12us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.628962 (+ 531us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.629022 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":88,"threads_started":1}
I20260429 20:00:45.629325 3980 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.624403 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.27.17.193:33871)
0429 20:00:45.624765 (+ 362us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.624770 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.624864 (+ 94us) server_negotiation.cc:408] Connection header received
0429 20:00:45.625073 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.625078 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.625289 (+ 211us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.625429 (+ 140us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:45.625830 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.626834 (+ 1004us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.628329 (+ 1495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.628853 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.629013 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.629086 (+ 73us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.629162 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260429 20:00:45.631443 3887 tablet_service.cc:1511] Processing CreateTablet for tablet 37992d5f2c5f4e7da1679d3481c90ba5 (DEFAULT_TABLE table=test-table [id=2537b19eabf149a6865e94257838671b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:45.631942 3603 tablet_service.cc:1511] Processing CreateTablet for tablet 37992d5f2c5f4e7da1679d3481c90ba5 (DEFAULT_TABLE table=test-table [id=2537b19eabf149a6865e94257838671b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:45.631942 3744 tablet_service.cc:1511] Processing CreateTablet for tablet 37992d5f2c5f4e7da1679d3481c90ba5 (DEFAULT_TABLE table=test-table [id=2537b19eabf149a6865e94257838671b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:45.632392 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37992d5f2c5f4e7da1679d3481c90ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:45.632809 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37992d5f2c5f4e7da1679d3481c90ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:45.632809 3603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37992d5f2c5f4e7da1679d3481c90ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:45.638048 3981 tablet_bootstrap.cc:492] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Bootstrap starting.
I20260429 20:00:45.639219 3982 tablet_bootstrap.cc:492] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Bootstrap starting.
I20260429 20:00:45.639328 3983 tablet_bootstrap.cc:492] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Bootstrap starting.
I20260429 20:00:45.640949 3981 tablet_bootstrap.cc:654] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:45.641294 3983 tablet_bootstrap.cc:654] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:45.641292 3982 tablet_bootstrap.cc:654] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:45.642036 3981 log.cc:826] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:45.642129 3982 log.cc:826] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:45.642421 3983 log.cc:826] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:45.643895 3981 tablet_bootstrap.cc:492] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: No bootstrap required, opened a new log
I20260429 20:00:45.644062 3982 tablet_bootstrap.cc:492] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: No bootstrap required, opened a new log
I20260429 20:00:45.644062 3983 tablet_bootstrap.cc:492] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: No bootstrap required, opened a new log
I20260429 20:00:45.644136 3981 ts_tablet_manager.cc:1403] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 20:00:45.644256 3982 ts_tablet_manager.cc:1403] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260429 20:00:45.644248 3983 ts_tablet_manager.cc:1403] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260429 20:00:45.646668 3981 raft_consensus.cc:359] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.646852 3981 raft_consensus.cc:385] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:45.646886 3981 raft_consensus.cc:740] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Initialized, Role: FOLLOWER
I20260429 20:00:45.647289 3983 raft_consensus.cc:359] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.647289 3982 raft_consensus.cc:359] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.647493 3982 raft_consensus.cc:385] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:45.647547 3982 raft_consensus.cc:740] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cb678006b45e585f008d4627b5cb0, State: Initialized, Role: FOLLOWER
I20260429 20:00:45.647473 3981 consensus_queue.cc:260] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.647650 3983 raft_consensus.cc:385] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:45.647984 3982 consensus_queue.cc:260] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.647722 3983 raft_consensus.cc:740] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Initialized, Role: FOLLOWER
I20260429 20:00:45.648408 3953 heartbeater.cc:499] Master 127.27.17.254:36801 was elected leader, sending a full tablet report...
I20260429 20:00:45.648617 3981 ts_tablet_manager.cc:1434] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260429 20:00:45.648729 3983 consensus_queue.cc:260] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.649154 3669 heartbeater.cc:499] Master 127.27.17.254:36801 was elected leader, sending a full tablet report...
I20260429 20:00:45.649371 3982 ts_tablet_manager.cc:1434] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260429 20:00:45.649606 3983 ts_tablet_manager.cc:1434] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260429 20:00:45.649917 3811 heartbeater.cc:499] Master 127.27.17.254:36801 was elected leader, sending a full tablet report...
I20260429 20:00:45.783838 3987 raft_consensus.cc:493] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:45.784060 3987 raft_consensus.cc:515] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.785238 3987 leader_election.cc:290] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141), 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871)
I20260429 20:00:45.788915 3846 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.785594 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:33871 (local address 127.27.17.195:58359)
0429 20:00:45.785740 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.785757 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.785864 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.786174 (+ 310us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.786178 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.786217 (+ 39us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:45.786466 (+ 249us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.786501 (+ 35us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.787546 (+ 1045us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.787549 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.788436 (+ 887us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.788445 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.788596 (+ 151us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.788620 (+ 24us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.788675 (+ 55us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.788744 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"spinlock_wait_cycles":31872}
I20260429 20:00:45.788931 3990 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.785590 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:38141 (local address 127.27.17.195:59041)
0429 20:00:45.786064 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:45.786082 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:45.786218 (+ 136us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:45.786530 (+ 312us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:45.786536 (+ 6us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:45.786563 (+ 27us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:45.786789 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.786794 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.787743 (+ 949us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.787747 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:45.788551 (+ 804us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:45.788559 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.788648 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.788664 (+ 16us) client_negotiation.cc:770] Sending connection context
0429 20:00:45.788710 (+ 46us) client_negotiation.cc:241] Negotiation successful
0429 20:00:45.788748 (+ 38us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":138,"spinlock_wait_cycles":2944,"thread_start_us":143,"threads_started":1}
I20260429 20:00:45.789526 3976 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.785790 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:59041 (local address 127.27.17.194:38141)
0429 20:00:45.785920 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.785925 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.786202 (+ 277us) server_negotiation.cc:408] Connection header received
0429 20:00:45.786382 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.786385 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.786427 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.786494 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:45.786932 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.787631 (+ 699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.788684 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.789204 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.789256 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.789321 (+ 65us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.789390 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 20:00:45.790020 3980 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.785790 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:58359 (local address 127.27.17.193:33871)
0429 20:00:45.785920 (+ 130us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.785925 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.785942 (+ 17us) server_negotiation.cc:408] Connection header received
0429 20:00:45.786008 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.786010 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.786053 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.786129 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:45.786642 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.787370 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.788632 (+ 1262us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.789737 (+ 1105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.789772 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.789834 (+ 62us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.789884 (+ 50us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260429 20:00:45.790248 3761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37992d5f2c5f4e7da1679d3481c90ba5" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0117ac2a4f834ed18442f237d154d200" is_pre_election: true
I20260429 20:00:45.790575 3761 raft_consensus.cc:2468] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 0.
I20260429 20:00:45.790655 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37992d5f2c5f4e7da1679d3481c90ba5" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0" is_pre_election: true
I20260429 20:00:45.790987 3623 raft_consensus.cc:2468] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 0.
I20260429 20:00:45.791106 3839 leader_election.cc:304] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:45.791429 3987 raft_consensus.cc:2804] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:45.791513 3987 raft_consensus.cc:493] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:45.791555 3987 raft_consensus.cc:3060] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:45.792533 3987 raft_consensus.cc:515] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.792900 3987 leader_election.cc:290] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 election: Requested vote from peers 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141), 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871)
I20260429 20:00:45.793413 3761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37992d5f2c5f4e7da1679d3481c90ba5" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0117ac2a4f834ed18442f237d154d200"
I20260429 20:00:45.793418 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37992d5f2c5f4e7da1679d3481c90ba5" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0"
I20260429 20:00:45.793527 3761 raft_consensus.cc:3060] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:45.793588 3623 raft_consensus.cc:3060] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:45.794513 3761 raft_consensus.cc:2468] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 1.
I20260429 20:00:45.794634 3623 raft_consensus.cc:2468] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 1.
I20260429 20:00:45.794843 3839 leader_election.cc:304] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:45.795053 3987 raft_consensus.cc:2804] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:45.795303 3987 raft_consensus.cc:697] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 LEADER]: Becoming Leader. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Running, Role: LEADER
I20260429 20:00:45.795619 3987 consensus_queue.cc:237] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } }
I20260429 20:00:45.799078 3464 catalog_manager.cc:5671] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195). New cstate: current_term: 1 leader_uuid: "706bab00e1d34a7592c131fb3fca0e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } health_report { overall_health: UNKNOWN } } }
W20260429 20:00:45.803876 3812 tablet.cc:2404] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 20:00:45.805569 3670 tablet.cc:2404] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260429 20:00:45.812947 3954 tablet.cc:2404] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260429 20:00:45.823377 3978 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:45.820034 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.27.17.195:42599)
0429 20:00:45.820170 (+ 136us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:45.820173 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:45.820187 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:45.820293 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:45.820296 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:45.820350 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:45.820436 (+ 86us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:45.820802 (+ 366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.821336 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:45.822123 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:45.822301 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:45.822355 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:45.822764 (+ 409us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:45.822848 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:45.823052 (+ 204us) server_negotiation.cc:300] Negotiation successful
0429 20:00:45.823099 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260429 20:00:45.829087 3623 raft_consensus.cc:1275] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Refusing update from remote peer 706bab00e1d34a7592c131fb3fca0e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:45.829090 3761 raft_consensus.cc:1275] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Refusing update from remote peer 706bab00e1d34a7592c131fb3fca0e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:45.829828 3991 consensus_queue.cc:1048] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:45.830015 3987 consensus_queue.cc:1048] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:45.837765 3995 mvcc.cc:204] Tried to move back new op lower bound from 7280610696508383232 to 7280610696381849600. Current Snapshot: MvccSnapshot[applied={T|T < 7280610696508383232}]
I20260429 20:00:45.838449 3996 mvcc.cc:204] Tried to move back new op lower bound from 7280610696508383232 to 7280610696381849600. Current Snapshot: MvccSnapshot[applied={T|T < 7280610696508383232}]
I20260429 20:00:45.845672 3462 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:45.845840 3462 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:45.848318 3462 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Sending DeleteTablet for 3 replicas of tablet 37992d5f2c5f4e7da1679d3481c90ba5
I20260429 20:00:45.848985 3744 tablet_service.cc:1558] Processing DeleteTablet for tablet 37992d5f2c5f4e7da1679d3481c90ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:45 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:57988
I20260429 20:00:45.849185 3603 tablet_service.cc:1558] Processing DeleteTablet for tablet 37992d5f2c5f4e7da1679d3481c90ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:45 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260429 20:00:45.849300 3887 tablet_service.cc:1558] Processing DeleteTablet for tablet 37992d5f2c5f4e7da1679d3481c90ba5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:45 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:44802
I20260429 20:00:45.849673 4004 tablet_replica.cc:333] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: stopping tablet replica
I20260429 20:00:45.849771 4005 tablet_replica.cc:333] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: stopping tablet replica
I20260429 20:00:45.849985 4005 raft_consensus.cc:2243] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:00:45.849987 4004 raft_consensus.cc:2243] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:45.850284 4004 raft_consensus.cc:2272] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:45.850447 4005 raft_consensus.cc:2272] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:45.850459 3462 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:45.850963 3462 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:45.851835 4003 tablet_replica.cc:333] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: stopping tablet replica
I20260429 20:00:45.851888 4005 ts_tablet_manager.cc:1916] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:45.852118 4003 raft_consensus.cc:2243] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:45.852439 4003 raft_consensus.cc:2272] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:45.853816 4003 ts_tablet_manager.cc:1916] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:45.856557 4003 ts_tablet_manager.cc:1929] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:45.856706 4003 log.cc:1199] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/37992d5f2c5f4e7da1679d3481c90ba5
I20260429 20:00:45.857045 4003 ts_tablet_manager.cc:1950] T 37992d5f2c5f4e7da1679d3481c90ba5 P 0117ac2a4f834ed18442f237d154d200: Deleting consensus metadata
I20260429 20:00:45.857160 3603 tablet_service.cc:1511] Processing CreateTablet for tablet 905c75319ae94b7cbe13547703abd5d7 (DEFAULT_TABLE table=test-table [id=69f27e04407c444d8f1d3de80eae3f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:45.857359 4005 ts_tablet_manager.cc:1929] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:45.857445 4005 log.cc:1199] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/37992d5f2c5f4e7da1679d3481c90ba5
I20260429 20:00:45.857486 3603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905c75319ae94b7cbe13547703abd5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:45.857756 4005 ts_tablet_manager.cc:1950] T 37992d5f2c5f4e7da1679d3481c90ba5 P 706bab00e1d34a7592c131fb3fca0e59: Deleting consensus metadata
I20260429 20:00:45.858179 4004 ts_tablet_manager.cc:1916] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:45.859740 3450 catalog_manager.cc:5002] TS 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599): tablet 37992d5f2c5f4e7da1679d3481c90ba5 (table test-table [id=2537b19eabf149a6865e94257838671b]) successfully deleted
I20260429 20:00:45.859824 3982 tablet_bootstrap.cc:492] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Bootstrap starting.
I20260429 20:00:45.859990 3887 tablet_service.cc:1511] Processing CreateTablet for tablet 905c75319ae94b7cbe13547703abd5d7 (DEFAULT_TABLE table=test-table [id=69f27e04407c444d8f1d3de80eae3f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:45.860196 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905c75319ae94b7cbe13547703abd5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:45.860265 4004 ts_tablet_manager.cc:1929] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:45.860348 4004 log.cc:1199] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/37992d5f2c5f4e7da1679d3481c90ba5
I20260429 20:00:45.860668 4004 ts_tablet_manager.cc:1950] T 37992d5f2c5f4e7da1679d3481c90ba5 P 214cb678006b45e585f008d4627b5cb0: Deleting consensus metadata
I20260429 20:00:45.860962 3982 tablet_bootstrap.cc:654] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:45.862076 3981 tablet_bootstrap.cc:492] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Bootstrap starting.
I20260429 20:00:45.862317 3744 tablet_service.cc:1511] Processing CreateTablet for tablet 905c75319ae94b7cbe13547703abd5d7 (DEFAULT_TABLE table=test-table [id=69f27e04407c444d8f1d3de80eae3f58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:45.862529 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 905c75319ae94b7cbe13547703abd5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:45.863093 3981 tablet_bootstrap.cc:654] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:45.863718 3450 catalog_manager.cc:5002] TS 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871): tablet 37992d5f2c5f4e7da1679d3481c90ba5 (table test-table [id=2537b19eabf149a6865e94257838671b]) successfully deleted
I20260429 20:00:45.864465 3981 tablet_bootstrap.cc:492] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: No bootstrap required, opened a new log
I20260429 20:00:45.864554 3981 ts_tablet_manager.cc:1403] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:45.864646 3983 tablet_bootstrap.cc:492] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Bootstrap starting.
I20260429 20:00:45.864758 3449 catalog_manager.cc:5002] TS 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141): tablet 37992d5f2c5f4e7da1679d3481c90ba5 (table test-table [id=2537b19eabf149a6865e94257838671b]) successfully deleted
I20260429 20:00:45.865026 3981 raft_consensus.cc:359] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:45.865145 3981 raft_consensus.cc:385] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:45.865180 3981 raft_consensus.cc:740] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Initialized, Role: FOLLOWER
I20260429 20:00:45.865314 3981 consensus_queue.cc:260] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:45.865612 3981 ts_tablet_manager.cc:1434] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 20:00:45.865654 3983 tablet_bootstrap.cc:654] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:45.866266 3982 tablet_bootstrap.cc:492] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: No bootstrap required, opened a new log
I20260429 20:00:45.866365 3982 ts_tablet_manager.cc:1403] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260429 20:00:45.866981 3983 tablet_bootstrap.cc:492] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: No bootstrap required, opened a new log
I20260429 20:00:45.866859 3982 raft_consensus.cc:359] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:45.867074 3983 ts_tablet_manager.cc:1403] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:45.867095 3982 raft_consensus.cc:385] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:45.867134 3982 raft_consensus.cc:740] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cb678006b45e585f008d4627b5cb0, State: Initialized, Role: FOLLOWER
I20260429 20:00:45.867250 3982 consensus_queue.cc:260] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:45.867496 3982 ts_tablet_manager.cc:1434] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:45.867584 3983 raft_consensus.cc:359] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:45.867663 3983 raft_consensus.cc:385] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:45.867684 3983 raft_consensus.cc:740] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Initialized, Role: FOLLOWER
I20260429 20:00:45.867777 3983 consensus_queue.cc:260] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:45.868031 3983 ts_tablet_manager.cc:1434] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:46.164994 3987 raft_consensus.cc:493] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:46.165187 3987 raft_consensus.cc:515] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.165926 3987 leader_election.cc:290] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871), 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141)
I20260429 20:00:46.166597 3761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "905c75319ae94b7cbe13547703abd5d7" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0117ac2a4f834ed18442f237d154d200" is_pre_election: true
I20260429 20:00:46.166700 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "905c75319ae94b7cbe13547703abd5d7" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0" is_pre_election: true
I20260429 20:00:46.166774 3761 raft_consensus.cc:2468] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 0.
I20260429 20:00:46.166867 3623 raft_consensus.cc:2468] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 0.
I20260429 20:00:46.167191 3839 leader_election.cc:304] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:46.167416 3987 raft_consensus.cc:2804] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:46.167495 3987 raft_consensus.cc:493] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:46.167596 3987 raft_consensus.cc:3060] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.168663 3987 raft_consensus.cc:515] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.169070 3987 leader_election.cc:290] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 election: Requested vote from peers 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871), 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141)
I20260429 20:00:46.169605 3761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "905c75319ae94b7cbe13547703abd5d7" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0117ac2a4f834ed18442f237d154d200"
I20260429 20:00:46.169677 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "905c75319ae94b7cbe13547703abd5d7" candidate_uuid: "706bab00e1d34a7592c131fb3fca0e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0"
I20260429 20:00:46.169790 3761 raft_consensus.cc:3060] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.169818 3623 raft_consensus.cc:3060] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.170715 3623 raft_consensus.cc:2468] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 1.
I20260429 20:00:46.170845 3761 raft_consensus.cc:2468] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 706bab00e1d34a7592c131fb3fca0e59 in term 1.
I20260429 20:00:46.171161 3840 leader_election.cc:304] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214cb678006b45e585f008d4627b5cb0, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:46.171379 3987 raft_consensus.cc:2804] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:46.171489 3987 raft_consensus.cc:697] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 LEADER]: Becoming Leader. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Running, Role: LEADER
I20260429 20:00:46.171751 3987 consensus_queue.cc:237] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.173710 3464 catalog_manager.cc:5671] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195). New cstate: current_term: 1 leader_uuid: "706bab00e1d34a7592c131fb3fca0e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } health_report { overall_health: HEALTHY } } }
Apr 29 20:00:46 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492844, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.254@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:46.261303 4015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.252909 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:36801 (local address 127.0.0.1:45660)
0429 20:00:46.253243 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:46.253257 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:46.253398 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:46.253734 (+ 336us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:46.253737 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:46.254136 (+ 399us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:46.254367 (+ 231us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.254373 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.255224 (+ 851us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.255227 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:46.255631 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.255637 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.255732 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.256382 (+ 650us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:46.256402 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:46.257956 (+ 1554us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:46.259895 (+ 1939us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:46.259903 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:46.259907 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:46.260138 (+ 231us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:46.260393 (+ 255us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:46.260396 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:46.260397 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:46.260452 (+ 55us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:46.260841 (+ 389us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:46.260846 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:46.260941 (+ 95us) client_negotiation.cc:770] Sending connection context
0429 20:00:46.261033 (+ 92us) client_negotiation.cc:241] Negotiation successful
0429 20:00:46.261157 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":131,"threads_started":1}
I20260429 20:00:46.261442 4016 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.253021 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45660 (local address 127.27.17.254:36801)
0429 20:00:46.253278 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.253284 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.253363 (+ 79us) server_negotiation.cc:408] Connection header received
0429 20:00:46.253552 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.253555 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.253600 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.253688 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:46.254513 (+ 825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.255120 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.255751 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.255897 (+ 146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.258087 (+ 2190us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:46.258106 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:46.258109 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:46.258132 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:46.259771 (+ 1639us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:46.260241 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:46.260244 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:46.260245 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:46.260286 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:46.260573 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:46.260576 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:46.260577 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:46.260736 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:46.260828 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.261127 (+ 299us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.261230 (+ 103us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":93,"threads_started":1}
I20260429 20:00:46.266076 3464 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:45660:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260429 20:00:46.266826 3464 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:46.272782 3603 tablet_service.cc:1511] Processing CreateTablet for tablet 8a9c104c622a4a7094191110916680ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edba26546c6c4ed6a6e3abd971668e02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 20:00:46.272827 3744 tablet_service.cc:1511] Processing CreateTablet for tablet 8a9c104c622a4a7094191110916680ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edba26546c6c4ed6a6e3abd971668e02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 20:00:46.273196 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9c104c622a4a7094191110916680ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.273200 3603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9c104c622a4a7094191110916680ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.273903 3887 tablet_service.cc:1511] Processing CreateTablet for tablet 8a9c104c622a4a7094191110916680ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=edba26546c6c4ed6a6e3abd971668e02]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260429 20:00:46.274240 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a9c104c622a4a7094191110916680ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.275746 3983 tablet_bootstrap.cc:492] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: Bootstrap starting.
I20260429 20:00:46.275934 3982 tablet_bootstrap.cc:492] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: Bootstrap starting.
I20260429 20:00:46.276829 3981 tablet_bootstrap.cc:492] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: Bootstrap starting.
I20260429 20:00:46.276898 3983 tablet_bootstrap.cc:654] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.276898 3982 tablet_bootstrap.cc:654] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.277760 3981 tablet_bootstrap.cc:654] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.278165 3983 tablet_bootstrap.cc:492] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: No bootstrap required, opened a new log
I20260429 20:00:46.278206 3982 tablet_bootstrap.cc:492] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: No bootstrap required, opened a new log
I20260429 20:00:46.278277 3983 ts_tablet_manager.cc:1403] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:46.278311 3982 ts_tablet_manager.cc:1403] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:46.278731 3983 raft_consensus.cc:359] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.278752 3982 raft_consensus.cc:359] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.278872 3982 raft_consensus.cc:385] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.278862 3983 raft_consensus.cc:385] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.278926 3983 raft_consensus.cc:740] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.278905 3982 raft_consensus.cc:740] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cb678006b45e585f008d4627b5cb0, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.278990 3981 tablet_bootstrap.cc:492] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: No bootstrap required, opened a new log
I20260429 20:00:46.279062 3981 ts_tablet_manager.cc:1403] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:46.279052 3983 consensus_queue.cc:260] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.279066 3982 consensus_queue.cc:260] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.279337 3983 ts_tablet_manager.cc:1434] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:46.279359 3982 ts_tablet_manager.cc:1434] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260429 20:00:46.279417 3989 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.279476 3981 raft_consensus.cc:359] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.279556 3989 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.279592 3981 raft_consensus.cc:385] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.279628 3981 raft_consensus.cc:740] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.279620 3988 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.279724 3988 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.279743 3981 consensus_queue.cc:260] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.280016 3981 ts_tablet_manager.cc:1434] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:46.280243 3987 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.280355 3987 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.424976 3989 raft_consensus.cc:493] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:46.425138 3989 raft_consensus.cc:515] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.426220 3989 leader_election.cc:290] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871), 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:46.429656 4022 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.426967 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:42599 (local address 127.27.17.194:38559)
0429 20:00:46.427233 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:46.427247 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:46.427342 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:46.427643 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:46.427645 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:46.427661 (+ 16us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:46.427862 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.427867 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.428678 (+ 811us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.428681 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:46.429240 (+ 559us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.429245 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.429383 (+ 138us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.429403 (+ 20us) client_negotiation.cc:770] Sending connection context
0429 20:00:46.429456 (+ 53us) client_negotiation.cc:241] Negotiation successful
0429 20:00:46.429514 (+ 58us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":52,"threads_started":1}
I20260429 20:00:46.429724 4020 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.426578 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:33871 (local address 127.27.17.194:53511)
0429 20:00:46.426869 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:46.426939 (+ 70us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:46.427103 (+ 164us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:46.427541 (+ 438us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:46.427548 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:46.427573 (+ 25us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:46.427855 (+ 282us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.427862 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.428644 (+ 782us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.428649 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:46.429253 (+ 604us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.429261 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.429413 (+ 152us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.429430 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 20:00:46.429519 (+ 89us) client_negotiation.cc:241] Negotiation successful
0429 20:00:46.429574 (+ 55us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
I20260429 20:00:46.430404 4021 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.426658 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:53511 (local address 127.27.17.193:33871)
0429 20:00:46.426976 (+ 318us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.426981 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.427272 (+ 291us) server_negotiation.cc:408] Connection header received
0429 20:00:46.427335 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.427338 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.427397 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.427491 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:46.427985 (+ 494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.428514 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.429415 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.430160 (+ 745us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.430205 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.430267 (+ 62us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.430310 (+ 43us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":77,"threads_started":1}
I20260429 20:00:46.430791 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a9c104c622a4a7094191110916680ae" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0" is_pre_election: true
I20260429 20:00:46.430838 4023 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.427112 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:38559 (local address 127.27.17.195:42599)
0429 20:00:46.427369 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.427372 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.427386 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:46.427433 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.427436 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.427513 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.427617 (+ 104us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:46.427988 (+ 371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.428533 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.429378 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.430572 (+ 1194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.430608 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.430666 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.430735 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260429 20:00:46.430981 3623 raft_consensus.cc:2468] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 0.
I20260429 20:00:46.431387 3907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a9c104c622a4a7094191110916680ae" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706bab00e1d34a7592c131fb3fca0e59" is_pre_election: true
I20260429 20:00:46.431406 3698 leader_election.cc:304] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 214cb678006b45e585f008d4627b5cb0; no voters:
I20260429 20:00:46.431615 3907 raft_consensus.cc:2468] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 0.
I20260429 20:00:46.431677 3989 raft_consensus.cc:2804] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:46.431761 3989 raft_consensus.cc:493] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:46.431811 3989 raft_consensus.cc:3060] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.432735 3989 raft_consensus.cc:515] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.433115 3989 leader_election.cc:290] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 election: Requested vote from peers 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871), 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:46.433781 3907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a9c104c622a4a7094191110916680ae" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706bab00e1d34a7592c131fb3fca0e59"
I20260429 20:00:46.433781 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a9c104c622a4a7094191110916680ae" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0"
I20260429 20:00:46.433914 3907 raft_consensus.cc:3060] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.433914 3623 raft_consensus.cc:3060] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.435055 3623 raft_consensus.cc:2468] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 1.
I20260429 20:00:46.435055 3907 raft_consensus.cc:2468] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 1.
I20260429 20:00:46.435379 3698 leader_election.cc:304] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 214cb678006b45e585f008d4627b5cb0; no voters:
I20260429 20:00:46.435561 3989 raft_consensus.cc:2804] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:46.435766 3989 raft_consensus.cc:697] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [term 1 LEADER]: Becoming Leader. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Running, Role: LEADER
I20260429 20:00:46.436024 3989 consensus_queue.cc:237] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.437280 4024 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: TxnStatusTablet state changed. Reason: New leader 0117ac2a4f834ed18442f237d154d200. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.437435 4024 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: This TxnStatusTablet replica's current role is: LEADER
I20260429 20:00:46.437767 4026 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20260429 20:00:46.437901 4026 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20260429 20:00:46.438868 3463 catalog_manager.cc:5671] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0117ac2a4f834ed18442f237d154d200 (127.27.17.194). New cstate: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:46.476788 4029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.472997 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58000 (local address 127.27.17.194:38141)
0429 20:00:46.473831 (+ 834us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.473834 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.473846 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:46.473918 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.473926 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.473977 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.474100 (+ 123us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:46.474448 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.475020 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.475723 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.475942 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.475993 (+ 51us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.476348 (+ 355us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.476466 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.476578 (+ 112us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.476626 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":631,"threads_started":1}
I20260429 20:00:46.476768 4015 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.472865 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:38141 (local address 127.0.0.1:58000)
0429 20:00:46.473006 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:46.473029 (+ 23us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:46.473136 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:46.474117 (+ 981us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:46.474121 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:46.474131 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 20:00:46.474329 (+ 198us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.474335 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.475148 (+ 813us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.475152 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:46.475594 (+ 442us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.475601 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.475732 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.475776 (+ 44us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.476455 (+ 679us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.476468 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 20:00:46.476564 (+ 96us) client_negotiation.cc:241] Negotiation successful
0429 20:00:46.476642 (+ 78us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260429 20:00:46.480067 3623 raft_consensus.cc:1275] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Refusing update from remote peer 0117ac2a4f834ed18442f237d154d200: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:46.480177 3907 raft_consensus.cc:1275] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Refusing update from remote peer 0117ac2a4f834ed18442f237d154d200: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:46.480530 3989 consensus_queue.cc:1048] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:46.480731 4024 consensus_queue.cc:1048] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200 [LEADER]: Connected to new peer: Peer: permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:46.482112 3988 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: TxnStatusTablet state changed. Reason: New leader 0117ac2a4f834ed18442f237d154d200. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.482210 3988 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.482451 3987 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: TxnStatusTablet state changed. Reason: New leader 0117ac2a4f834ed18442f237d154d200. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.482568 3987 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.484068 3989 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.484179 3989 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: This TxnStatusTablet replica's current role is: LEADER
I20260429 20:00:46.484838 3989 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.484930 3989 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 0117ac2a4f834ed18442f237d154d200: This TxnStatusTablet replica's current role is: LEADER
I20260429 20:00:46.485102 3988 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.485203 3988 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 214cb678006b45e585f008d4627b5cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.487331 3987 tablet_replica.cc:442] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } } }
I20260429 20:00:46.487437 3987 tablet_replica.cc:445] T 8a9c104c622a4a7094191110916680ae P 706bab00e1d34a7592c131fb3fca0e59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260429 20:00:46.488107 4033 mvcc.cc:204] Tried to move back new op lower bound from 7280610699176755200 to 7280610699004678144. Current Snapshot: MvccSnapshot[applied={T|T < 7280610699176755200}]
I20260429 20:00:46.500259 4040 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.496697 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:38141 (local address 127.27.17.195:45471)
0429 20:00:46.496979 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:46.496993 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:46.497084 (+ 91us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:46.497363 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:46.497366 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:46.497375 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 20:00:46.497599 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.497606 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.498686 (+ 1080us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.498689 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:46.499276 (+ 587us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.499287 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.499412 (+ 125us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.499450 (+ 38us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.499982 (+ 532us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.499997 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 20:00:46.500067 (+ 70us) client_negotiation.cc:241] Negotiation successful
0429 20:00:46.500129 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260429 20:00:46.500289 4029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.496804 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:45471 (local address 127.27.17.194:38141)
0429 20:00:46.496938 (+ 134us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.496943 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.497064 (+ 121us) server_negotiation.cc:408] Connection header received
0429 20:00:46.497178 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.497181 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.497239 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.497326 (+ 87us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:46.497722 (+ 396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.498552 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.499408 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.499636 (+ 228us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.499685 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.499779 (+ 94us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.499862 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.500102 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.500153 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260429 20:00:46.507712 3623 raft_consensus.cc:1275] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Refusing update from remote peer 706bab00e1d34a7592c131fb3fca0e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:46.508137 3761 raft_consensus.cc:1275] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Refusing update from remote peer 706bab00e1d34a7592c131fb3fca0e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:46.508172 3991 consensus_queue.cc:1048] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:46.508661 3991 consensus_queue.cc:1048] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:46.530995 4023 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.527593 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:58161 (local address 127.27.17.195:42599)
0429 20:00:46.527735 (+ 142us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.527739 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.527753 (+ 14us) server_negotiation.cc:408] Connection header received
0429 20:00:46.527797 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.527801 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.527853 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.527936 (+ 83us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:46.528588 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.529295 (+ 707us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.530309 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.530458 (+ 149us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.530496 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.530570 (+ 74us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.530629 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.530787 (+ 158us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.530827 (+ 40us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 20:00:46.531000 4045 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.527237 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:42599 (local address 127.27.17.194:58161)
0429 20:00:46.527528 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:46.527546 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:46.527646 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:46.527959 (+ 313us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:46.527962 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:46.527972 (+ 10us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 20:00:46.528446 (+ 474us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.528455 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.529507 (+ 1052us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.529511 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:46.530188 (+ 677us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:46.530196 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.530336 (+ 140us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.530379 (+ 43us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.530662 (+ 283us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.530668 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 20:00:46.530740 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 20:00:46.530802 (+ 62us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":109,"threads_started":1}
I20260429 20:00:46.556052 3462 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:46.556226 3462 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:46.558419 3462 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Sending DeleteTablet for 3 replicas of tablet 905c75319ae94b7cbe13547703abd5d7
I20260429 20:00:46.559087 3744 tablet_service.cc:1558] Processing DeleteTablet for tablet 905c75319ae94b7cbe13547703abd5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:46 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:57988
I20260429 20:00:46.562458 4048 tablet_replica.cc:333] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: stopping tablet replica
I20260429 20:00:46.562623 4048 raft_consensus.cc:2243] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:46.562738 3603 tablet_service.cc:1558] Processing DeleteTablet for tablet 905c75319ae94b7cbe13547703abd5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:46 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260429 20:00:46.562826 4048 raft_consensus.cc:2272] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:46.563558 3887 tablet_service.cc:1558] Processing DeleteTablet for tablet 905c75319ae94b7cbe13547703abd5d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:46 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:44802
I20260429 20:00:46.563653 4048 ts_tablet_manager.cc:1916] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:46.567268 4048 ts_tablet_manager.cc:1929] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 20:00:46.567425 4048 log.cc:1199] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/905c75319ae94b7cbe13547703abd5d7
I20260429 20:00:46.567823 4048 ts_tablet_manager.cc:1950] T 905c75319ae94b7cbe13547703abd5d7 P 0117ac2a4f834ed18442f237d154d200: Deleting consensus metadata
I20260429 20:00:46.568739 3449 catalog_manager.cc:5002] TS 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141): tablet 905c75319ae94b7cbe13547703abd5d7 (table test-table [id=69f27e04407c444d8f1d3de80eae3f58]) successfully deleted
I20260429 20:00:46.568890 4050 tablet_replica.cc:333] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: stopping tablet replica
I20260429 20:00:46.569084 4051 tablet_replica.cc:333] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: stopping tablet replica
I20260429 20:00:46.569259 4051 raft_consensus.cc:2243] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:00:46.569448 4050 raft_consensus.cc:2243] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:46.569746 4051 raft_consensus.cc:2272] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:46.569921 4050 raft_consensus.cc:2272] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:46.570590 4050 ts_tablet_manager.cc:1916] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:46.571815 4051 ts_tablet_manager.cc:1916] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:46.574261 4051 ts_tablet_manager.cc:1929] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 20:00:46.574260 4050 ts_tablet_manager.cc:1929] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 20:00:46.574337 4050 log.cc:1199] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/905c75319ae94b7cbe13547703abd5d7
I20260429 20:00:46.574401 4051 log.cc:1199] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/905c75319ae94b7cbe13547703abd5d7
I20260429 20:00:46.574673 4050 ts_tablet_manager.cc:1950] T 905c75319ae94b7cbe13547703abd5d7 P 214cb678006b45e585f008d4627b5cb0: Deleting consensus metadata
I20260429 20:00:46.574734 4051 ts_tablet_manager.cc:1950] T 905c75319ae94b7cbe13547703abd5d7 P 706bab00e1d34a7592c131fb3fca0e59: Deleting consensus metadata
I20260429 20:00:46.575533 3450 catalog_manager.cc:5002] TS 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599): tablet 905c75319ae94b7cbe13547703abd5d7 (table test-table [id=69f27e04407c444d8f1d3de80eae3f58]) successfully deleted
I20260429 20:00:46.575738 3450 catalog_manager.cc:5002] TS 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871): tablet 905c75319ae94b7cbe13547703abd5d7 (table test-table [id=69f27e04407c444d8f1d3de80eae3f58]) successfully deleted
Apr 29 20:00:46 dist-test-slave-01nr krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Apr 29 20:00:46 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:46 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 20:00:46.595131 4016 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:46.588984 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.27.17.254:36801)
0429 20:00:46.589135 (+ 151us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.589139 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.589400 (+ 261us) server_negotiation.cc:408] Connection header received
0429 20:00:46.589592 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.589596 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.589650 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.589743 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:46.590659 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.591255 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.591928 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.592142 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.594999 (+ 2857us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45662: BlockingRecv error: recv got EOF from 127.0.0.1:45662 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 20:00:46 dist-test-slave-01nr krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM
I20260429 20:00:46.604426 4016 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.596634 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.27.17.254:36801)
0429 20:00:46.596755 (+ 121us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.596759 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.596854 (+ 95us) server_negotiation.cc:408] Connection header received
0429 20:00:46.597003 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.597007 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.597053 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.597149 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:46.597857 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.598356 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.599047 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.599261 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.601257 (+ 1996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:46.601279 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:46.601281 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:46.601309 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:46.602849 (+ 1540us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:46.603308 (+ 459us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:46.603312 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:46.603314 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:46.603367 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:46.603620 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:46.603623 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:46.603625 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:46.603774 (+ 149us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:46.603915 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.604134 (+ 219us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.604254 (+ 120us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 20:00:46.607563 3462 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:00:46.608068 3462 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:46.613276 3603 tablet_service.cc:1511] Processing CreateTablet for tablet 4c97990e23494d5892ad52d9a8c69a43 (DEFAULT_TABLE table=test-table [id=7c6687c36b114b9d9b7c9b8edf743324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:46.613619 3603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c97990e23494d5892ad52d9a8c69a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.613696 3744 tablet_service.cc:1511] Processing CreateTablet for tablet 4c97990e23494d5892ad52d9a8c69a43 (DEFAULT_TABLE table=test-table [id=7c6687c36b114b9d9b7c9b8edf743324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:46.613920 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c97990e23494d5892ad52d9a8c69a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.615999 3983 tablet_bootstrap.cc:492] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Bootstrap starting.
I20260429 20:00:46.616068 3982 tablet_bootstrap.cc:492] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Bootstrap starting.
I20260429 20:00:46.616842 3983 tablet_bootstrap.cc:654] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.617031 3982 tablet_bootstrap.cc:654] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.617847 3887 tablet_service.cc:1511] Processing CreateTablet for tablet 4c97990e23494d5892ad52d9a8c69a43 (DEFAULT_TABLE table=test-table [id=7c6687c36b114b9d9b7c9b8edf743324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:46.618144 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c97990e23494d5892ad52d9a8c69a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.619534 3983 tablet_bootstrap.cc:492] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: No bootstrap required, opened a new log
I20260429 20:00:46.619633 3983 ts_tablet_manager.cc:1403] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260429 20:00:46.620121 3983 raft_consensus.cc:359] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.620229 3981 tablet_bootstrap.cc:492] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Bootstrap starting.
I20260429 20:00:46.620236 3983 raft_consensus.cc:385] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.620282 3983 raft_consensus.cc:740] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.620397 3983 consensus_queue.cc:260] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.620671 3983 ts_tablet_manager.cc:1434] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 20:00:46.621157 3981 tablet_bootstrap.cc:654] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.621529 3982 tablet_bootstrap.cc:492] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: No bootstrap required, opened a new log
I20260429 20:00:46.621608 3982 ts_tablet_manager.cc:1403] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260429 20:00:46.622025 3982 raft_consensus.cc:359] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.622128 3982 raft_consensus.cc:385] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.622155 3982 raft_consensus.cc:740] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cb678006b45e585f008d4627b5cb0, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.622272 3982 consensus_queue.cc:260] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.622495 3982 ts_tablet_manager.cc:1434] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260429 20:00:46.623652 3981 tablet_bootstrap.cc:492] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: No bootstrap required, opened a new log
I20260429 20:00:46.623744 3981 ts_tablet_manager.cc:1403] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260429 20:00:46.624238 3981 raft_consensus.cc:359] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.624346 3981 raft_consensus.cc:385] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.624380 3981 raft_consensus.cc:740] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.624495 3981 consensus_queue.cc:260] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.624806 3981 ts_tablet_manager.cc:1434] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:46.880486 4024 raft_consensus.cc:493] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:46.880661 4024 raft_consensus.cc:515] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.881336 4024 leader_election.cc:290] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871), 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:46.881848 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c97990e23494d5892ad52d9a8c69a43" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0" is_pre_election: true
I20260429 20:00:46.881904 3907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c97990e23494d5892ad52d9a8c69a43" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706bab00e1d34a7592c131fb3fca0e59" is_pre_election: true
I20260429 20:00:46.882019 3623 raft_consensus.cc:2468] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 0.
I20260429 20:00:46.882059 3907 raft_consensus.cc:2468] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 0.
I20260429 20:00:46.882417 3698 leader_election.cc:304] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:46.882651 4024 raft_consensus.cc:2804] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:46.882718 4024 raft_consensus.cc:493] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:46.882766 4024 raft_consensus.cc:3060] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.883826 4024 raft_consensus.cc:515] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.884207 4024 leader_election.cc:290] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 election: Requested vote from peers 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871), 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:46.884786 3907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c97990e23494d5892ad52d9a8c69a43" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706bab00e1d34a7592c131fb3fca0e59"
I20260429 20:00:46.884776 3623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4c97990e23494d5892ad52d9a8c69a43" candidate_uuid: "0117ac2a4f834ed18442f237d154d200" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "214cb678006b45e585f008d4627b5cb0"
I20260429 20:00:46.884908 3623 raft_consensus.cc:3060] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.884908 3907 raft_consensus.cc:3060] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:46.886059 3907 raft_consensus.cc:2468] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 1.
I20260429 20:00:46.886059 3623 raft_consensus.cc:2468] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0117ac2a4f834ed18442f237d154d200 in term 1.
I20260429 20:00:46.886389 3698 leader_election.cc:304] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:46.886646 4024 raft_consensus.cc:2804] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:46.886765 4024 raft_consensus.cc:697] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 1 LEADER]: Becoming Leader. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Running, Role: LEADER
I20260429 20:00:46.886986 4024 consensus_queue.cc:237] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.888851 3462 catalog_manager.cc:5671] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0117ac2a4f834ed18442f237d154d200 (127.27.17.194). New cstate: current_term: 1 leader_uuid: "0117ac2a4f834ed18442f237d154d200" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:46.915644 4029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:46.912854 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.27.17.194:38141)
0429 20:00:46.912974 (+ 120us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:46.912978 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:46.912991 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:46.913072 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:46.913075 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:46.913134 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:46.913201 (+ 67us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:46.913639 (+ 438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.914124 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:46.914868 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:46.915088 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:46.915129 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:46.915207 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:46.915274 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:46.915392 (+ 118us) server_negotiation.cc:300] Negotiation successful
0429 20:00:46.915436 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260429 20:00:46.918170 3623 raft_consensus.cc:1275] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Refusing update from remote peer 0117ac2a4f834ed18442f237d154d200: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:46.918146 3907 raft_consensus.cc:1275] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Refusing update from remote peer 0117ac2a4f834ed18442f237d154d200: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:46.918784 4024 consensus_queue.cc:1048] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [LEADER]: Connected to new peer: Peer: permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:46.919025 3989 consensus_queue.cc:1048] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [LEADER]: Connected to new peer: Peer: permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:46.929246 3462 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:46.929433 3462 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:46.931895 3462 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Sending DeleteTablet for 3 replicas of tablet 4c97990e23494d5892ad52d9a8c69a43
I20260429 20:00:46.932591 3744 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c97990e23494d5892ad52d9a8c69a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:46 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:57988
I20260429 20:00:46.932642 3603 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c97990e23494d5892ad52d9a8c69a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:46 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260429 20:00:46.932783 4048 tablet_replica.cc:333] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: stopping tablet replica
I20260429 20:00:46.932792 3887 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c97990e23494d5892ad52d9a8c69a43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:46 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:44802
I20260429 20:00:46.932840 4050 tablet_replica.cc:333] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: stopping tablet replica
I20260429 20:00:46.932905 4048 raft_consensus.cc:2243] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:00:46.932955 4050 raft_consensus.cc:2243] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:46.933080 4050 raft_consensus.cc:2272] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:46.933187 4048 raft_consensus.cc:2272] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:46.933408 4051 tablet_replica.cc:333] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: stopping tablet replica
I20260429 20:00:46.933502 4051 raft_consensus.cc:2243] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:46.933604 4051 raft_consensus.cc:2272] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:46.934262 3462 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260429 20:00:46.934469 4050 ts_tablet_manager.cc:1916] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260429 20:00:46.934875 3462 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:00:46.935220 4048 ts_tablet_manager.cc:1916] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:46.936889 4050 ts_tablet_manager.cc:1929] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:46.936997 4050 log.cc:1199] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/4c97990e23494d5892ad52d9a8c69a43
I20260429 20:00:46.937304 4050 ts_tablet_manager.cc:1950] T 4c97990e23494d5892ad52d9a8c69a43 P 214cb678006b45e585f008d4627b5cb0: Deleting consensus metadata
I20260429 20:00:46.937763 4048 ts_tablet_manager.cc:1929] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:46.937920 4048 log.cc:1199] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/4c97990e23494d5892ad52d9a8c69a43
I20260429 20:00:46.938145 3450 catalog_manager.cc:5002] TS 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871): tablet 4c97990e23494d5892ad52d9a8c69a43 (table test-table [id=7c6687c36b114b9d9b7c9b8edf743324]) successfully deleted
I20260429 20:00:46.938299 4048 ts_tablet_manager.cc:1950] T 4c97990e23494d5892ad52d9a8c69a43 P 0117ac2a4f834ed18442f237d154d200: Deleting consensus metadata
I20260429 20:00:46.939433 3449 catalog_manager.cc:5002] TS 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141): tablet 4c97990e23494d5892ad52d9a8c69a43 (table test-table [id=7c6687c36b114b9d9b7c9b8edf743324]) successfully deleted
I20260429 20:00:46.939977 4051 ts_tablet_manager.cc:1916] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:46.941138 3603 tablet_service.cc:1511] Processing CreateTablet for tablet 164a319a4bfb4b2aaa8719964894fe09 (DEFAULT_TABLE table=test-table [id=eba10386f57b4a7198f0c88d6de5b79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:46.941351 3887 tablet_service.cc:1511] Processing CreateTablet for tablet 164a319a4bfb4b2aaa8719964894fe09 (DEFAULT_TABLE table=test-table [id=eba10386f57b4a7198f0c88d6de5b79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:46.941444 3603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164a319a4bfb4b2aaa8719964894fe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.941591 3887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164a319a4bfb4b2aaa8719964894fe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.942358 4051 ts_tablet_manager.cc:1929] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:00:46.942421 4051 log.cc:1199] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/4c97990e23494d5892ad52d9a8c69a43
I20260429 20:00:46.942692 4051 ts_tablet_manager.cc:1950] T 4c97990e23494d5892ad52d9a8c69a43 P 706bab00e1d34a7592c131fb3fca0e59: Deleting consensus metadata
I20260429 20:00:46.943701 3981 tablet_bootstrap.cc:492] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Bootstrap starting.
I20260429 20:00:46.943874 3982 tablet_bootstrap.cc:492] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Bootstrap starting.
I20260429 20:00:46.944399 3981 tablet_bootstrap.cc:654] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.944903 3982 tablet_bootstrap.cc:654] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.945060 3450 catalog_manager.cc:5002] TS 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599): tablet 4c97990e23494d5892ad52d9a8c69a43 (table test-table [id=7c6687c36b114b9d9b7c9b8edf743324]) successfully deleted
I20260429 20:00:46.945648 3981 tablet_bootstrap.cc:492] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: No bootstrap required, opened a new log
I20260429 20:00:46.945755 3981 ts_tablet_manager.cc:1403] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260429 20:00:46.946238 3981 raft_consensus.cc:359] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.946354 3981 raft_consensus.cc:385] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.946394 3981 raft_consensus.cc:740] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 706bab00e1d34a7592c131fb3fca0e59, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.946542 3981 consensus_queue.cc:260] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.946625 3744 tablet_service.cc:1511] Processing CreateTablet for tablet 164a319a4bfb4b2aaa8719964894fe09 (DEFAULT_TABLE table=test-table [id=eba10386f57b4a7198f0c88d6de5b79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:00:46.946825 3981 ts_tablet_manager.cc:1434] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:46.946936 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164a319a4bfb4b2aaa8719964894fe09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:46.949170 3983 tablet_bootstrap.cc:492] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Bootstrap starting.
I20260429 20:00:46.949613 3982 tablet_bootstrap.cc:492] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: No bootstrap required, opened a new log
I20260429 20:00:46.949725 3982 ts_tablet_manager.cc:1403] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260429 20:00:46.949976 3983 tablet_bootstrap.cc:654] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:46.950222 3982 raft_consensus.cc:359] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.950340 3982 raft_consensus.cc:385] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.950376 3982 raft_consensus.cc:740] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214cb678006b45e585f008d4627b5cb0, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.950513 3982 consensus_queue.cc:260] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.951174 3983 tablet_bootstrap.cc:492] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: No bootstrap required, opened a new log
I20260429 20:00:46.951278 3983 ts_tablet_manager.cc:1403] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:46.951758 3983 raft_consensus.cc:359] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.951858 3983 raft_consensus.cc:385] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:46.951892 3983 raft_consensus.cc:740] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0117ac2a4f834ed18442f237d154d200, State: Initialized, Role: FOLLOWER
I20260429 20:00:46.951898 3982 ts_tablet_manager.cc:1434] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260429 20:00:46.952003 3983 consensus_queue.cc:260] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:46.952289 3983 ts_tablet_manager.cc:1434] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260429 20:00:47.129309 3988 raft_consensus.cc:493] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:00:47.129524 3988 raft_consensus.cc:515] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:47.130648 3988 leader_election.cc:290] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141), 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:47.134644 4076 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.130943 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:38141 (local address 127.27.17.193:59345)
0429 20:00:47.131217 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:47.131234 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:47.131380 (+ 146us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:47.131729 (+ 349us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:47.131732 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:47.131761 (+ 29us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:47.132073 (+ 312us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.132081 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.133163 (+ 1082us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.133167 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:47.134170 (+ 1003us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.134179 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.134307 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.134325 (+ 18us) client_negotiation.cc:770] Sending connection context
0429 20:00:47.134404 (+ 79us) client_negotiation.cc:241] Negotiation successful
0429 20:00:47.134448 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"spinlock_wait_cycles":36992,"thread_start_us":94,"threads_started":1}
I20260429 20:00:47.134653 4075 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.130949 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:42599 (local address 127.27.17.193:43479)
0429 20:00:47.131242 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:47.131255 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:47.131380 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:47.131729 (+ 349us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:47.131732 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:47.131759 (+ 27us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:00:47.132073 (+ 314us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.132081 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.133163 (+ 1082us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.133167 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:47.134170 (+ 1003us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.134179 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.134308 (+ 129us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.134325 (+ 17us) client_negotiation.cc:770] Sending connection context
0429 20:00:47.134383 (+ 58us) client_negotiation.cc:241] Negotiation successful
0429 20:00:47.134448 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":15,"spinlock_wait_cycles":44032,"thread_start_us":78,"threads_started":1}
I20260429 20:00:47.135157 4029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.131016 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:59345 (local address 127.27.17.194:38141)
0429 20:00:47.131285 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:47.131290 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:47.131309 (+ 19us) server_negotiation.cc:408] Connection header received
0429 20:00:47.131537 (+ 228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:47.131541 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:47.131599 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:47.131694 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:47.132235 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.133021 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.134333 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.134856 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.134935 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:47.134983 (+ 48us) server_negotiation.cc:300] Negotiation successful
0429 20:00:47.135044 (+ 61us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260429 20:00:47.135175 4077 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.131068 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:43479 (local address 127.27.17.195:42599)
0429 20:00:47.131308 (+ 240us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:47.131312 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:47.131327 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:47.131537 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:47.131541 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:47.131599 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:47.131694 (+ 95us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:00:47.132235 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.133022 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.134333 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.134856 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.134925 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:47.134984 (+ 59us) server_negotiation.cc:300] Negotiation successful
0429 20:00:47.135058 (+ 74us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
I20260429 20:00:47.135653 3761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "164a319a4bfb4b2aaa8719964894fe09" candidate_uuid: "214cb678006b45e585f008d4627b5cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0117ac2a4f834ed18442f237d154d200" is_pre_election: true
I20260429 20:00:47.135654 3907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "164a319a4bfb4b2aaa8719964894fe09" candidate_uuid: "214cb678006b45e585f008d4627b5cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706bab00e1d34a7592c131fb3fca0e59" is_pre_election: true
I20260429 20:00:47.135816 3907 raft_consensus.cc:2468] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214cb678006b45e585f008d4627b5cb0 in term 0.
I20260429 20:00:47.135816 3761 raft_consensus.cc:2468] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 214cb678006b45e585f008d4627b5cb0 in term 0.
I20260429 20:00:47.136222 3555 leader_election.cc:304] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0117ac2a4f834ed18442f237d154d200, 214cb678006b45e585f008d4627b5cb0, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:47.136574 3988 raft_consensus.cc:2804] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:00:47.136636 3988 raft_consensus.cc:493] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:00:47.136665 3988 raft_consensus.cc:3060] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:47.137661 3988 raft_consensus.cc:515] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:47.138036 3988 leader_election.cc:290] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [CANDIDATE]: Term 1 election: Requested vote from peers 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141), 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599)
I20260429 20:00:47.138417 3907 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "164a319a4bfb4b2aaa8719964894fe09" candidate_uuid: "214cb678006b45e585f008d4627b5cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "706bab00e1d34a7592c131fb3fca0e59"
I20260429 20:00:47.138521 3907 raft_consensus.cc:3060] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:47.138464 3761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "164a319a4bfb4b2aaa8719964894fe09" candidate_uuid: "214cb678006b45e585f008d4627b5cb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0117ac2a4f834ed18442f237d154d200"
I20260429 20:00:47.138692 3761 raft_consensus.cc:3060] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:47.139609 3907 raft_consensus.cc:2468] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214cb678006b45e585f008d4627b5cb0 in term 1.
I20260429 20:00:47.139701 3761 raft_consensus.cc:2468] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 214cb678006b45e585f008d4627b5cb0 in term 1.
I20260429 20:00:47.140017 3556 leader_election.cc:304] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 214cb678006b45e585f008d4627b5cb0, 706bab00e1d34a7592c131fb3fca0e59; no voters:
I20260429 20:00:47.140281 3988 raft_consensus.cc:2804] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:47.140457 3988 raft_consensus.cc:697] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 1 LEADER]: Becoming Leader. State: Replica: 214cb678006b45e585f008d4627b5cb0, State: Running, Role: LEADER
I20260429 20:00:47.140700 3988 consensus_queue.cc:237] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } }
I20260429 20:00:47.143162 3462 catalog_manager.cc:5671] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 214cb678006b45e585f008d4627b5cb0 (127.27.17.193). New cstate: current_term: 1 leader_uuid: "214cb678006b45e585f008d4627b5cb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "214cb678006b45e585f008d4627b5cb0" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 33871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 } health_report { overall_health: UNKNOWN } } }
I20260429 20:00:47.204417 4080 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.201212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.27.17.193:33871)
0429 20:00:47.201529 (+ 317us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:47.201532 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:47.201545 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:47.201618 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:47.201620 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:47.201671 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:47.201777 (+ 106us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:47.202090 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.202703 (+ 613us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.203412 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.203589 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.203643 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:47.204037 (+ 394us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:47.204126 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:47.204228 (+ 102us) server_negotiation.cc:300] Negotiation successful
0429 20:00:47.204274 (+ 46us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":117,"threads_started":1}
I20260429 20:00:47.215687 4029 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.212751 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:54645 (local address 127.27.17.194:38141)
0429 20:00:47.212909 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:47.212913 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:47.213141 (+ 228us) server_negotiation.cc:408] Connection header received
0429 20:00:47.213242 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:47.213245 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:47.213297 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:47.213377 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:47.213828 (+ 451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.214296 (+ 468us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.215050 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.215187 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.215223 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:47.215302 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:47.215373 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:47.215521 (+ 148us) server_negotiation.cc:300] Negotiation successful
0429 20:00:47.215575 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260429 20:00:47.215708 4083 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.212683 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:38141 (local address 127.27.17.193:54645)
0429 20:00:47.213012 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:47.213034 (+ 22us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:47.213154 (+ 120us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:47.213432 (+ 278us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:47.213436 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:47.213444 (+ 8us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 20:00:47.213698 (+ 254us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.213704 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.214428 (+ 724us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.214431 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:47.214925 (+ 494us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.214932 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.215040 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.215077 (+ 37us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 20:00:47.215415 (+ 338us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 20:00:47.215422 (+ 7us) client_negotiation.cc:770] Sending connection context
0429 20:00:47.215494 (+ 72us) client_negotiation.cc:241] Negotiation successful
0429 20:00:47.215570 (+ 76us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":134,"threads_started":1}
I20260429 20:00:47.222275 3907 raft_consensus.cc:1275] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Refusing update from remote peer 214cb678006b45e585f008d4627b5cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:47.222301 3761 raft_consensus.cc:1275] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Refusing update from remote peer 214cb678006b45e585f008d4627b5cb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:00:47.222851 3988 consensus_queue.cc:1048] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "706bab00e1d34a7592c131fb3fca0e59" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 42599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:47.223151 4078 consensus_queue.cc:1048] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0117ac2a4f834ed18442f237d154d200" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 38141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:00:47.243953 4080 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.240905 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:52735 (local address 127.27.17.193:33871)
0429 20:00:47.241037 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:47.241043 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:47.241298 (+ 255us) server_negotiation.cc:408] Connection header received
0429 20:00:47.241353 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:47.241359 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:47.241427 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:47.241526 (+ 99us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:00:47.241883 (+ 357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.242415 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.243123 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.243370 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.243420 (+ 50us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:00:47.243498 (+ 78us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:00:47.243580 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:47.243788 (+ 208us) server_negotiation.cc:300] Negotiation successful
0429 20:00:47.243842 (+ 54us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 20:00:47.243973 4088 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.240861 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:33871 (local address 127.27.17.194:52735)
0429 20:00:47.241100 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:47.241115 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:47.241218 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:47.241552 (+ 334us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:47.241555 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:47.241564 (+ 9us) client_negotiation.cc:190] Negotiated authn=TOKEN
0429 20:00:47.241771 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.241777 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.242541 (+ 764us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:47.242551 (+ 10us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:47.243013 (+ 462us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:47.243020 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:47.243138 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:47.243163 (+ 25us) client_negotiation.cc:253] Sending TOKEN_EXCHANGE NegotiatePB request
0429 20:00:47.243694 (+ 531us) client_negotiation.cc:272] Received TOKEN_EXCHANGE NegotiatePB response
0429 20:00:47.243700 (+ 6us) client_negotiation.cc:770] Sending connection context
0429 20:00:47.243756 (+ 56us) client_negotiation.cc:241] Negotiation successful
0429 20:00:47.243808 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260429 20:00:47.268935 3463 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:47.269151 3463 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:00:47.271797 3463 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 2f8efd6d15fb45aaa8a4a38ab53aeecc: Sending DeleteTablet for 3 replicas of tablet 164a319a4bfb4b2aaa8719964894fe09
I20260429 20:00:47.272830 3744 tablet_service.cc:1558] Processing DeleteTablet for tablet 164a319a4bfb4b2aaa8719964894fe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:47 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:57988
I20260429 20:00:47.272837 3603 tablet_service.cc:1558] Processing DeleteTablet for tablet 164a319a4bfb4b2aaa8719964894fe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:47 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:40578
I20260429 20:00:47.272858 3887 tablet_service.cc:1558] Processing DeleteTablet for tablet 164a319a4bfb4b2aaa8719964894fe09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:00:47 UTC) from {username='oryx', principal='oryx/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:44802
I20260429 20:00:47.273041 4048 tablet_replica.cc:333] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: stopping tablet replica
I20260429 20:00:47.273039 4050 tablet_replica.cc:333] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: stopping tablet replica
I20260429 20:00:47.273169 4048 raft_consensus.cc:2243] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:47.273169 4050 raft_consensus.cc:2243] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:00:47.273236 4051 tablet_replica.cc:333] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: stopping tablet replica
I20260429 20:00:47.273298 4048 raft_consensus.cc:2272] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:47.273368 4051 raft_consensus.cc:2243] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:00:47.273505 4051 raft_consensus.cc:2272] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:47.273506 4050 raft_consensus.cc:2272] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:00:47.274176 4048 ts_tablet_manager.cc:1916] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:47.274267 4050 ts_tablet_manager.cc:1916] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:47.274376 4051 ts_tablet_manager.cc:1916] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:00:47.276536 4050 ts_tablet_manager.cc:1929] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 20:00:47.276536 4048 ts_tablet_manager.cc:1929] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 20:00:47.276610 4050 log.cc:1199] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-0/wal/wals/164a319a4bfb4b2aaa8719964894fe09
I20260429 20:00:47.276638 4048 log.cc:1199] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-1/wal/wals/164a319a4bfb4b2aaa8719964894fe09
I20260429 20:00:47.276880 4048 ts_tablet_manager.cc:1950] T 164a319a4bfb4b2aaa8719964894fe09 P 0117ac2a4f834ed18442f237d154d200: Deleting consensus metadata
I20260429 20:00:47.276880 4050 ts_tablet_manager.cc:1950] T 164a319a4bfb4b2aaa8719964894fe09 P 214cb678006b45e585f008d4627b5cb0: Deleting consensus metadata
I20260429 20:00:47.276937 4051 ts_tablet_manager.cc:1929] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260429 20:00:47.277012 4051 log.cc:1199] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/164a319a4bfb4b2aaa8719964894fe09
I20260429 20:00:47.277402 4051 ts_tablet_manager.cc:1950] T 164a319a4bfb4b2aaa8719964894fe09 P 706bab00e1d34a7592c131fb3fca0e59: Deleting consensus metadata
I20260429 20:00:47.277665 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 3538
I20260429 20:00:47.278215 3450 catalog_manager.cc:5002] TS 706bab00e1d34a7592c131fb3fca0e59 (127.27.17.195:42599): tablet 164a319a4bfb4b2aaa8719964894fe09 (table test-table [id=eba10386f57b4a7198f0c88d6de5b79d]) successfully deleted
I20260429 20:00:47.279394 3449 catalog_manager.cc:5002] TS 0117ac2a4f834ed18442f237d154d200 (127.27.17.194:38141): tablet 164a319a4bfb4b2aaa8719964894fe09 (table test-table [id=eba10386f57b4a7198f0c88d6de5b79d]) successfully deleted
W20260429 20:00:47.284925 3450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.17.193:33871 (error 108)
I20260429 20:00:47.285208 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 3680
I20260429 20:00:47.287271 4092 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:47.286803 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:33871 (local address 127.0.0.1:40606)
0429 20:00:47.287112 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:47.287171 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.193:33871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
W20260429 20:00:47.287544 3450 catalog_manager.cc:4729] TS 214cb678006b45e585f008d4627b5cb0 (127.27.17.193:33871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 164a319a4bfb4b2aaa8719964894fe09: Network error: Client connection negotiation failed: client connection to 127.27.17.193:33871: connect: Connection refused (error 111)
I20260429 20:00:47.293100 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 3822
I20260429 20:00:47.300034 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 3432
2026-04-29T20:00:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5002 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:47 dist-test-slave-01nr krb5kdc[4095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:47 dist-test-slave-01nr krb5kdc[4095](info): set up 2 sockets
Apr 29 20:00:47 dist-test-slave-01nr krb5kdc[4095](info): commencing operation
krb5kdc: starting...
W20260429 20:00:49.328359 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.004s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:49 dist-test-slave-01nr krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:49.492230 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:42441
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:42441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:49.600140 4111 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:49.600425 4111 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:49.600538 4111 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:49.604168 4111 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:49.604251 4111 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:49.604275 4111 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:49.604300 4111 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:49.604365 4111 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:49.608879 4111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:42441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:42441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:49.610044 4111 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:49.610960 4111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:49.616613 4117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:49.616752 4119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:49.616706 4111 server_base.cc:1061] running on GCE node
W20260429 20:00:49.616765 4116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:49.617377 4111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:49.618372 4111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:49.619604 4111 hybrid_clock.cc:648] HybridClock initialized: now 1777492849619574 us; error 44 us; skew 500 ppm
W20260429 20:00:49.622870 4111 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-04-29T20:00:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2328 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:00:49 dist-test-slave-01nr krb5kdc[4124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:00:49 dist-test-slave-01nr krb5kdc[4124](info): set up 2 sockets
Apr 29 20:00:49 dist-test-slave-01nr krb5kdc[4124](info): commencing operation
krb5kdc: starting...
W20260429 20:00:51.649252 27719 mini_kdc.cc:121] Time spent starting KDC: real 1.997s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:00:51 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:00:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:51.803429 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37961
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:51.908345 4140 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:51.908656 4140 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:51.908764 4140 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:51.912322 4140 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:51.912458 4140 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:51.912503 4140 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:51.912544 4140 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:51.912585 4140 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:51.917429 4140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:51.918704 4140 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:51.919677 4140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:51.925774 4148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:51.926021 4140 server_base.cc:1061] running on GCE node
W20260429 20:00:51.925774 4146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:51.925832 4145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:51.926712 4140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:51.927721 4140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:51.928896 4140 hybrid_clock.cc:648] HybridClock initialized: now 1777492851928869 us; error 42 us; skew 500 ppm
Apr 29 20:00:51 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492851, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:51.931661 4140 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:51.932847 4140 webserver.cc:492] Webserver started at http://127.27.17.254:46043/ using document root <none> and password file <none>
I20260429 20:00:51.933485 4140 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:51.933570 4140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:51.933825 4140 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:51.935786 4140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597"
format_stamp: "Formatted at 2026-04-29 20:00:51 on dist-test-slave-01nr"
server_key: "e3f63c7811c58de8115e59c5dbbd70ca"
server_key_iv: "1c306973abd674205a4c61335ff2f1b5"
server_key_version: "encryptionkey@0"
I20260429 20:00:51.936300 4140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597"
format_stamp: "Formatted at 2026-04-29 20:00:51 on dist-test-slave-01nr"
server_key: "e3f63c7811c58de8115e59c5dbbd70ca"
server_key_iv: "1c306973abd674205a4c61335ff2f1b5"
server_key_version: "encryptionkey@0"
I20260429 20:00:51.939863 4140 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:51.942543 4155 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:51.943580 4140 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:51.943732 4140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597"
format_stamp: "Formatted at 2026-04-29 20:00:51 on dist-test-slave-01nr"
server_key: "e3f63c7811c58de8115e59c5dbbd70ca"
server_key_iv: "1c306973abd674205a4c61335ff2f1b5"
server_key_version: "encryptionkey@0"
I20260429 20:00:51.943850 4140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:51.960352 4140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:51.963464 4140 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:51.963685 4140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:51.971310 4140 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:37961
I20260429 20:00:51.971328 4207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:37961 every 8 connection(s)
I20260429 20:00:51.972371 4140 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:51.975229 4208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:00:51.979919 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4140
I20260429 20:00:51.980067 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:00:51.980327 27719 external_mini_cluster.cc:1468] Setting key c9dc16523befa7c23b7473eff1975ae0
I20260429 20:00:51.980959 4208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Bootstrap starting.
I20260429 20:00:51.983280 4208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Neither blocks nor log segments found. Creating new log.
I20260429 20:00:51.984159 4208 log.cc:826] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:51.986172 4208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: No bootstrap required, opened a new log
Apr 29 20:00:51 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:51.989281 4208 raft_consensus.cc:359] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:51.989503 4208 raft_consensus.cc:385] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:00:51.989594 4208 raft_consensus.cc:740] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70e06a7ecc74d7596ddcb2bfdcbf597, State: Initialized, Role: FOLLOWER
I20260429 20:00:51.990087 4208 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:51.990247 4208 raft_consensus.cc:399] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:51.990316 4208 raft_consensus.cc:493] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:51.990410 4208 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:00:51.991531 4208 raft_consensus.cc:515] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:51.991925 4208 leader_election.cc:304] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e70e06a7ecc74d7596ddcb2bfdcbf597; no voters:
I20260429 20:00:51.992314 4208 leader_election.cc:290] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:00:51.992769 4213 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:00:51.992997 4213 raft_consensus.cc:697] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 LEADER]: Becoming Leader. State: Replica: e70e06a7ecc74d7596ddcb2bfdcbf597, State: Running, Role: LEADER
I20260429 20:00:51.993369 4213 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:51.993607 4208 sys_catalog.cc:565] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:51.994493 4211 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:51.981634 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.27.17.254:37961)
0429 20:00:51.982035 (+ 401us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:51.982044 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:51.982076 (+ 32us) server_negotiation.cc:408] Connection header received
0429 20:00:51.982617 (+ 541us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:51.982632 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:51.982957 (+ 325us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:51.983225 (+ 268us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:51.984358 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:51.985335 (+ 977us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:51.986189 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:51.986545 (+ 356us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:51.988952 (+ 2407us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:51.988975 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:51.988988 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:51.989020 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:51.991181 (+ 2161us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:51.991668 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:51.991674 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:51.991680 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:51.991769 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:51.992020 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:51.992027 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:51.992029 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:51.992541 (+ 512us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:51.992714 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:51.993477 (+ 763us) server_negotiation.cc:300] Negotiation successful
0429 20:00:51.993716 (+ 239us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":146,"threads_started":1}
I20260429 20:00:51.994952 4214 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } } }
I20260429 20:00:51.995033 4215 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70e06a7ecc74d7596ddcb2bfdcbf597. Latest consensus state: current_term: 1 leader_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } } }
I20260429 20:00:51.995090 4214 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:51.995496 4222 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:51.995092 4215 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:51.998672 4222 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:52.004186 4222 catalog_manager.cc:1357] Generated new cluster ID: 3a8461deced0412886d0b5c87f782281
I20260429 20:00:52.004278 4222 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:52.014269 4222 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:00:52.015491 4222 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:52.024392 4222 catalog_manager.cc:6044] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Generated new TSK 0
I20260429 20:00:52.025182 4222 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:52.087363 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37961
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:52.195390 4236 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:52.195644 4236 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:52.195747 4236 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:52.199162 4236 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:52.199263 4236 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:52.199383 4236 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:52.203804 4236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:52.204945 4236 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:52.205797 4236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:52.212715 4242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:52.212710 4241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:52.212710 4244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:52.212779 4236 server_base.cc:1061] running on GCE node
I20260429 20:00:52.213434 4236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:52.213991 4236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:52.215147 4236 hybrid_clock.cc:648] HybridClock initialized: now 1777492852215116 us; error 41 us; skew 500 ppm
Apr 29 20:00:52 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:52.217813 4236 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:00:52.218858 4236 webserver.cc:492] Webserver started at http://127.27.17.193:42801/ using document root <none> and password file <none>
I20260429 20:00:52.219475 4236 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:52.219527 4236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:52.219691 4236 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:52.221464 4236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "5f1f384e244c42f998813282364784ac"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "ca73b1bc91e9dfa8bb2e2a79607e386a"
server_key_iv: "286b4887e465e3d660726010d87535a8"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.221928 4236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "5f1f384e244c42f998813282364784ac"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "ca73b1bc91e9dfa8bb2e2a79607e386a"
server_key_iv: "286b4887e465e3d660726010d87535a8"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.225574 4236 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 20:00:52.227824 4251 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.228969 4236 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:52.229074 4236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "5f1f384e244c42f998813282364784ac"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "ca73b1bc91e9dfa8bb2e2a79607e386a"
server_key_iv: "286b4887e465e3d660726010d87535a8"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.229259 4236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:52.248492 4236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:52.251634 4236 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:52.251845 4236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:52.252475 4236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:52.253540 4236 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:52.253619 4236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.253681 4236 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:52.253716 4236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.264474 4236 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:37845
I20260429 20:00:52.264488 4364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:37845 every 8 connection(s)
I20260429 20:00:52.265547 4236 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:52.274317 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4236
I20260429 20:00:52.274451 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
Apr 29 20:00:52 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:52.274780 27719 external_mini_cluster.cc:1468] Setting key e0599b96bbc3f582910400534a541240
I20260429 20:00:52.279564 4211 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:52.267627 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:39495 (local address 127.27.17.254:37961)
0429 20:00:52.267785 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:52.267789 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:52.268405 (+ 616us) server_negotiation.cc:408] Connection header received
0429 20:00:52.269341 (+ 936us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:52.269345 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:52.269411 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:52.269518 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:52.271455 (+ 1937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.271996 (+ 541us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:52.272697 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.272881 (+ 184us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:52.275439 (+ 2558us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:52.275468 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:52.275470 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:52.275500 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:52.277492 (+ 1992us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:52.278032 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:52.278036 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:52.278038 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:52.278092 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:52.278413 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:52.278417 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:52.278419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:52.278584 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:52.278696 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:52.279262 (+ 566us) server_negotiation.cc:300] Negotiation successful
0429 20:00:52.279398 (+ 136us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 20:00:52.280206 4367 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:52.267795 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:39495)
0429 20:00:52.268234 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:52.268269 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:52.269107 (+ 838us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:52.269699 (+ 592us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:52.269708 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:52.270114 (+ 406us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:52.271205 (+ 1091us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:52.271228 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.272150 (+ 922us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:52.272154 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:52.272570 (+ 416us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:52.272577 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.272811 (+ 234us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:52.273506 (+ 695us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:52.273529 (+ 23us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:52.275236 (+ 1707us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:52.277634 (+ 2398us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:52.277641 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:52.277655 (+ 14us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:52.277919 (+ 264us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:52.278201 (+ 282us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:52.278204 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:52.278206 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:52.278314 (+ 108us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:52.278696 (+ 382us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:52.278704 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:52.279000 (+ 296us) client_negotiation.cc:770] Sending connection context
0429 20:00:52.279210 (+ 210us) client_negotiation.cc:241] Negotiation successful
0429 20:00:52.279452 (+ 242us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":100,"threads_started":1}
I20260429 20:00:52.281436 4365 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37961
I20260429 20:00:52.281716 4365 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:52.282249 4365 heartbeater.cc:507] Master 127.27.17.254:37961 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 20:00:52.283812 4172 ts_manager.cc:194] Registered new tserver with Master: 5f1f384e244c42f998813282364784ac (127.27.17.193:37845)
I20260429 20:00:52.285187 4172 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:39495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:52.326606 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37961
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:52.434034 4372 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:52.434278 4372 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:52.434338 4372 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:52.437870 4372 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:52.437955 4372 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:52.438038 4372 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:52.442435 4372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:52.443626 4372 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:52.444427 4372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260429 20:00:52.451220 4372 server_base.cc:1061] running on GCE node
W20260429 20:00:52.451143 4380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:52.451143 4378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:52.451143 4377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:52.452106 4372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:52.452787 4372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:52.453987 4372 hybrid_clock.cc:648] HybridClock initialized: now 1777492852453959 us; error 54 us; skew 500 ppm
Apr 29 20:00:52 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:52.457146 4372 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:00:52.458767 4372 webserver.cc:492] Webserver started at http://127.27.17.194:41415/ using document root <none> and password file <none>
I20260429 20:00:52.459890 4372 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:52.459996 4372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:52.460235 4372 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:52.462090 4372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "a4605d3669e84d0bac3adc1c05aeec04"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d7ffb95b5e0afa33c990eaa926ab8ba9"
server_key_iv: "e81affba9383dd95c371c3d89ce02cc2"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.462666 4372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "a4605d3669e84d0bac3adc1c05aeec04"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d7ffb95b5e0afa33c990eaa926ab8ba9"
server_key_iv: "e81affba9383dd95c371c3d89ce02cc2"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.466355 4372 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:52.468843 4387 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.470118 4372 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:52.470289 4372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "a4605d3669e84d0bac3adc1c05aeec04"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d7ffb95b5e0afa33c990eaa926ab8ba9"
server_key_iv: "e81affba9383dd95c371c3d89ce02cc2"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.470451 4372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:52.490094 4372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:52.493438 4372 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:52.493669 4372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:52.494261 4372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:52.495357 4372 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:52.495430 4372 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.495502 4372 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:52.495534 4372 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.506629 4372 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:38695
I20260429 20:00:52.506668 4500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:38695 every 8 connection(s)
I20260429 20:00:52.507722 4372 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:52.513350 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4372
I20260429 20:00:52.513459 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:00:52.513742 27719 external_mini_cluster.cc:1468] Setting key fdd593717420d019e3bac0830c81a183
Apr 29 20:00:52 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:52.522425 4211 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:52.509703 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:45423 (local address 127.27.17.254:37961)
0429 20:00:52.509852 (+ 149us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:52.509856 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:52.510512 (+ 656us) server_negotiation.cc:408] Connection header received
0429 20:00:52.511482 (+ 970us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:52.511486 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:52.511554 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:52.511668 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:52.513351 (+ 1683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.514193 (+ 842us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:52.515046 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.515271 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:52.518164 (+ 2893us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:52.518188 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:52.518190 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:52.518221 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:52.520168 (+ 1947us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:52.520861 (+ 693us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:52.520866 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:52.520867 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:52.520922 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:52.521263 (+ 341us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:52.521266 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:52.521268 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:52.521442 (+ 174us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:52.521563 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:52.522085 (+ 522us) server_negotiation.cc:300] Negotiation successful
0429 20:00:52.522249 (+ 164us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 20:00:52.523183 4503 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:52.509909 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:45423)
0429 20:00:52.510363 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:52.510399 (+ 36us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:52.511259 (+ 860us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:52.511840 (+ 581us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:52.511848 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:52.512255 (+ 407us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:52.513168 (+ 913us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:52.513183 (+ 15us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.514389 (+ 1206us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:52.514393 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:52.514867 (+ 474us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:52.514876 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.515172 (+ 296us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:52.515842 (+ 670us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:52.515866 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:52.517939 (+ 2073us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:52.520395 (+ 2456us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:52.520403 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:52.520418 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:52.520712 (+ 294us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:52.521026 (+ 314us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:52.521030 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:52.521035 (+ 5us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:52.521155 (+ 120us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:52.521564 (+ 409us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:52.521572 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:52.521823 (+ 251us) client_negotiation.cc:770] Sending connection context
0429 20:00:52.522037 (+ 214us) client_negotiation.cc:241] Negotiation successful
0429 20:00:52.522293 (+ 256us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260429 20:00:52.524566 4501 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37961
I20260429 20:00:52.524852 4501 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:52.525461 4501 heartbeater.cc:507] Master 127.27.17.254:37961 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 20:00:52.526844 4172 ts_manager.cc:194] Registered new tserver with Master: a4605d3669e84d0bac3adc1c05aeec04 (127.27.17.194:38695)
I20260429 20:00:52.527511 4172 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:45423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:00:52.576246 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:37961
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:52.689340 4508 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:52.689594 4508 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:52.689664 4508 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:52.693434 4508 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:52.693529 4508 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:52.693616 4508 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:52.698537 4508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:52.699800 4508 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:52.700717 4508 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:52.707654 4513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:52.707746 4516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:52.707767 4514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:52.707980 4508 server_base.cc:1061] running on GCE node
I20260429 20:00:52.708406 4508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:52.708999 4508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:52.710199 4508 hybrid_clock.cc:648] HybridClock initialized: now 1777492852710176 us; error 37 us; skew 500 ppm
Apr 29 20:00:52 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:52.713263 4508 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 20:00:52.714763 4508 webserver.cc:492] Webserver started at http://127.27.17.195:35873/ using document root <none> and password file <none>
I20260429 20:00:52.715438 4508 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:52.715530 4508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:52.715773 4508 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:00:52.717635 4508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "58a1b1cb4b8a46f29d1d0865d56db896"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d1f5cb2875f4be9989c836d65bc202c4"
server_key_iv: "e58d45a611366cfaa3e52294b39aa18e"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.718176 4508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "58a1b1cb4b8a46f29d1d0865d56db896"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d1f5cb2875f4be9989c836d65bc202c4"
server_key_iv: "e58d45a611366cfaa3e52294b39aa18e"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.722069 4508 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260429 20:00:52.725226 4523 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.726594 4508 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:52.726752 4508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "58a1b1cb4b8a46f29d1d0865d56db896"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d1f5cb2875f4be9989c836d65bc202c4"
server_key_iv: "e58d45a611366cfaa3e52294b39aa18e"
server_key_version: "encryptionkey@0"
I20260429 20:00:52.726881 4508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:52.744776 4508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:52.747974 4508 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:52.748206 4508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:52.748878 4508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:52.749820 4508 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:52.749893 4508 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.749965 4508 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:52.750008 4508 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:52.760040 4508 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:44113
I20260429 20:00:52.760058 4636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:44113 every 8 connection(s)
I20260429 20:00:52.761018 4508 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:52.763258 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4508
I20260429 20:00:52.763401 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:00:52.763701 27719 external_mini_cluster.cc:1468] Setting key fbdfe1025fde94b3a3e21cfc71e828ee
Apr 29 20:00:52 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492852, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:00:52.774108 4211 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:52.762850 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:39661 (local address 127.27.17.254:37961)
0429 20:00:52.763031 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:52.763037 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:52.763691 (+ 654us) server_negotiation.cc:408] Connection header received
0429 20:00:52.764657 (+ 966us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:52.764662 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:52.764728 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:52.764826 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:52.766830 (+ 2004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.767433 (+ 603us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:52.768084 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.768227 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:52.770422 (+ 2195us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:52.770452 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:52.770455 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:52.770485 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:52.771990 (+ 1505us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:52.772586 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:52.772590 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:52.772591 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:52.772657 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:52.773032 (+ 375us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:52.773035 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:52.773036 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:52.773212 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:52.773306 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:52.773849 (+ 543us) server_negotiation.cc:300] Negotiation successful
0429 20:00:52.773973 (+ 124us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260429 20:00:52.774804 4639 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:52.763110 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:39661)
0429 20:00:52.763530 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:52.763578 (+ 48us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:52.764452 (+ 874us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:52.764990 (+ 538us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:52.765002 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:52.765430 (+ 428us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:52.766615 (+ 1185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:52.766640 (+ 25us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.767564 (+ 924us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:52.767568 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:52.767970 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:52.767976 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:52.768139 (+ 163us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:52.768702 (+ 563us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:52.768721 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:52.770263 (+ 1542us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:00:52.772129 (+ 1866us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:52.772135 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:52.772145 (+ 10us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:52.772435 (+ 290us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:52.772788 (+ 353us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:00:52.772791 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:00:52.772793 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:00:52.772905 (+ 112us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:00:52.773358 (+ 453us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:00:52.773363 (+ 5us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:00:52.773603 (+ 240us) client_negotiation.cc:770] Sending connection context
0429 20:00:52.773785 (+ 182us) client_negotiation.cc:241] Negotiation successful
0429 20:00:52.773981 (+ 196us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260429 20:00:52.775943 4637 heartbeater.cc:344] Connected to a master server at 127.27.17.254:37961
I20260429 20:00:52.776181 4637 heartbeater.cc:461] Registering TS with master...
I20260429 20:00:52.776690 4637 heartbeater.cc:507] Master 127.27.17.254:37961 requested a full tablet report, sending...
I20260429 20:00:52.777833 4172 ts_manager.cc:194] Registered new tserver with Master: 58a1b1cb4b8a46f29d1d0865d56db896 (127.27.17.195:44113)
I20260429 20:00:52.778426 4172 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:39661
I20260429 20:00:52.790545 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.193.keytab.
Entry for principal oryx/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.193.keytab.
WARNING: no policy specified for oryx/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.194.keytab.
Entry for principal oryx/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.194.keytab.
WARNING: no policy specified for oryx/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.195.keytab.
Entry for principal oryx/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.195.keytab.
I20260429 20:00:52.875344 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4236
I20260429 20:00:52.881630 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4372
I20260429 20:00:52.887816 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4508
I20260429 20:00:52.893586 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4140
I20260429 20:00:52.900267 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37961
--webserver_interface=127.27.17.254
--webserver_port=46043
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:00:53.006680 4646 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:53.007076 4646 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:53.007210 4646 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:53.010782 4646 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:00:53.010886 4646 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:53.010955 4646 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:53.011001 4646 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:00:53.011044 4646 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:00:53.015689 4646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=46043
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:53.016884 4646 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:53.017756 4646 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:53.023308 4652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.023303 4651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.023370 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:53.024068 4646 server_base.cc:1061] running on GCE node
I20260429 20:00:53.024638 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:53.025601 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:53.026794 4646 hybrid_clock.cc:648] HybridClock initialized: now 1777492853026770 us; error 39 us; skew 500 ppm
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492853, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:53.029633 4646 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:00:53.030788 4646 webserver.cc:492] Webserver started at http://127.27.17.254:46043/ using document root <none> and password file <none>
I20260429 20:00:53.031387 4646 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:53.031479 4646 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:53.035058 4646 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260429 20:00:53.037174 4661 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.038311 4646 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:53.038460 4646 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597"
format_stamp: "Formatted at 2026-04-29 20:00:51 on dist-test-slave-01nr"
server_key: "e3f63c7811c58de8115e59c5dbbd70ca"
server_key_iv: "1c306973abd674205a4c61335ff2f1b5"
server_key_version: "encryptionkey@0"
I20260429 20:00:53.038898 4646 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:53.055027 4646 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:53.058332 4646 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:53.058588 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:53.066556 4646 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:37961
I20260429 20:00:53.066593 4713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:37961 every 8 connection(s)
I20260429 20:00:53.067709 4646 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:00:53.072340 4714 sys_catalog.cc:263] Verifying existing consensus state
I20260429 20:00:53.073901 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Bootstrap starting.
I20260429 20:00:53.076392 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4646
I20260429 20:00:53.076870 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:37845
--local_ip_for_outbound_sockets=127.27.17.193
--tserver_master_addrs=127.27.17.254:37961
--webserver_port=42801
--webserver_interface=127.27.17.193
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.17.193
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 20:00:53.091019 4714 log.cc:826] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Log is configured to *not* fsync() on all Append() calls
I20260429 20:00:53.097622 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260429 20:00:53.098066 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Bootstrap complete.
I20260429 20:00:53.101707 4714 raft_consensus.cc:359] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:53.102077 4714 raft_consensus.cc:740] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70e06a7ecc74d7596ddcb2bfdcbf597, State: Initialized, Role: FOLLOWER
I20260429 20:00:53.102615 4714 consensus_queue.cc:260] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:53.102774 4714 raft_consensus.cc:399] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:00:53.102851 4714 raft_consensus.cc:493] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:00:53.102988 4714 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 1 FOLLOWER]: Advancing to term 2
I20260429 20:00:53.104671 4714 raft_consensus.cc:515] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:53.105041 4714 leader_election.cc:304] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e70e06a7ecc74d7596ddcb2bfdcbf597; no voters:
I20260429 20:00:53.105558 4714 leader_election.cc:290] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260429 20:00:53.105706 4718 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 2 FOLLOWER]: Leader election won for term 2
I20260429 20:00:53.105967 4718 raft_consensus.cc:697] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [term 2 LEADER]: Becoming Leader. State: Replica: e70e06a7ecc74d7596ddcb2bfdcbf597, State: Running, Role: LEADER
I20260429 20:00:53.106287 4718 consensus_queue.cc:237] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } }
I20260429 20:00:53.106961 4714 sys_catalog.cc:565] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:00:53.107749 4719 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } } }
I20260429 20:00:53.107882 4719 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:53.108186 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e70e06a7ecc74d7596ddcb2bfdcbf597. Latest consensus state: current_term: 2 leader_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e70e06a7ecc74d7596ddcb2bfdcbf597" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 37961 } } }
I20260429 20:00:53.108263 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597 [sys.catalog]: This master's current role is: LEADER
I20260429 20:00:53.108801 4725 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:00:53.112154 4725 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:00:53.112852 4725 catalog_manager.cc:1269] Loaded cluster ID: 3a8461deced0412886d0b5c87f782281
I20260429 20:00:53.112911 4725 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:00:53.115166 4725 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:00:53.115875 4725 catalog_manager.cc:6055] T 00000000000000000000000000000000 P e70e06a7ecc74d7596ddcb2bfdcbf597: Loaded TSK: 0
I20260429 20:00:53.116849 4725 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260429 20:00:53.192710 4716 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:53.192950 4716 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:53.193050 4716 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:53.196596 4716 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:53.196678 4716 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:53.196830 4716 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:00:53.201450 4716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.193.keytab
--principal=oryx/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:37845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=42801
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:53.202692 4716 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:53.203579 4716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:53.210316 4740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.210369 4741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.210335 4743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:53.210809 4716 server_base.cc:1061] running on GCE node
I20260429 20:00:53.211241 4716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:53.211894 4716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:53.213073 4716 hybrid_clock.cc:648] HybridClock initialized: now 1777492853213064 us; error 37 us; skew 500 ppm
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492853, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:53.215960 4716 init.cc:377] Logged in from keytab as oryx/127.27.17.193@KRBTEST.COM (short username oryx)
I20260429 20:00:53.217247 4716 webserver.cc:492] Webserver started at http://127.27.17.193:42801/ using document root <none> and password file <none>
I20260429 20:00:53.217829 4716 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:53.217921 4716 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:53.221828 4716 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:00:53.223912 4750 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.225054 4716 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:53.225183 4716 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "5f1f384e244c42f998813282364784ac"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "ca73b1bc91e9dfa8bb2e2a79607e386a"
server_key_iv: "286b4887e465e3d660726010d87535a8"
server_key_version: "encryptionkey@0"
I20260429 20:00:53.225773 4716 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:53.236941 4716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:53.240010 4716 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:53.240232 4716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:53.240878 4716 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:53.241801 4716 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:53.241851 4716 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.241920 4716 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:53.241952 4716 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.254240 4716 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:37845
I20260429 20:00:53.254256 4863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:37845 every 8 connection(s)
I20260429 20:00:53.255429 4716 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:00:53.263808 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4716
I20260429 20:00:53.264307 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:38695
--local_ip_for_outbound_sockets=127.27.17.194
--tserver_master_addrs=127.27.17.254:37961
--webserver_port=41415
--webserver_interface=127.27.17.194
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.17.194
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 20:00:53.270635 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.257668 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:34225 (local address 127.27.17.254:37961)
0429 20:00:53.258274 (+ 606us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.258285 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.260426 (+ 2141us) server_negotiation.cc:408] Connection header received
0429 20:00:53.260733 (+ 307us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.260749 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.261099 (+ 350us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.261428 (+ 329us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.263351 (+ 1923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.265003 (+ 1652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.266312 (+ 1309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.266599 (+ 287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.269958 (+ 3359us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:34225: BlockingRecv error: recv got EOF from 127.27.17.193:34225 (error 108)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":122,"threads_started":1}
I20260429 20:00:53.270632 4867 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.257806 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:34225)
0429 20:00:53.259081 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.259126 (+ 45us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.260114 (+ 988us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.261746 (+ 1632us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.261757 (+ 11us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.262144 (+ 387us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.263147 (+ 1003us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.263166 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.265198 (+ 2032us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.265205 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.266137 (+ 932us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.266154 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.266447 (+ 293us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.267244 (+ 797us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.267270 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.269770 (+ 2500us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1058,"thread_start_us":124,"threads_started":1}
W20260429 20:00:53.271564 4864 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:37961 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.17.254:37961: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:53.277490 4867 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.272172 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:39899)
0429 20:00:53.272376 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.272391 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.272529 (+ 138us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.272818 (+ 289us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.272821 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.273020 (+ 199us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.273233 (+ 213us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.273239 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.274124 (+ 885us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.274130 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.274555 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.274569 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.274711 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.275283 (+ 572us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.275302 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.277343 (+ 2041us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20260429 20:00:53.277647 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.272240 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:39899 (local address 127.27.17.254:37961)
0429 20:00:53.272378 (+ 138us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.272385 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.272469 (+ 84us) server_negotiation.cc:408] Connection header received
0429 20:00:53.272657 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.272660 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.272709 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.272797 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.273368 (+ 571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.273970 (+ 602us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.274701 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.275266 (+ 565us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.277513 (+ 2247us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:39899: BlockingRecv error: recv got EOF from 127.27.17.193:39899 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:53.283432 4867 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.278326 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:60849)
0429 20:00:53.278506 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.278520 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.278621 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.278921 (+ 300us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.278925 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.279085 (+ 160us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.279249 (+ 164us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.279254 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.280002 (+ 748us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.280006 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.280380 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.280386 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.280516 (+ 130us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.281101 (+ 585us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.281115 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.283265 (+ 2150us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260429 20:00:53.283684 4864 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 20:00:53.283704 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.278391 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:60849 (local address 127.27.17.254:37961)
0429 20:00:53.278507 (+ 116us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.278512 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.278704 (+ 192us) server_negotiation.cc:408] Connection header received
0429 20:00:53.278743 (+ 39us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.278745 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.278789 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.278878 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.279374 (+ 496us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.279887 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.280499 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.280725 (+ 226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.283394 (+ 2669us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:60849: BlockingRecv error: recv got EOF from 127.27.17.193:60849 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260429 20:00:53.392321 4868 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:53.392561 4868 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:53.392622 4868 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:53.396319 4868 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:53.396421 4868 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:53.396509 4868 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:00:53.401114 4868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.194.keytab
--principal=oryx/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:38695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=41415
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:53.402215 4868 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:53.403054 4868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:53.410504 4876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.410497 4873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:53.410681 4868 server_base.cc:1061] running on GCE node
W20260429 20:00:53.410847 4874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:53.411295 4868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:53.411937 4868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:53.413148 4868 hybrid_clock.cc:648] HybridClock initialized: now 1777492853413121 us; error 40 us; skew 500 ppm
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492853, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:53.416134 4868 init.cc:377] Logged in from keytab as oryx/127.27.17.194@KRBTEST.COM (short username oryx)
I20260429 20:00:53.417363 4868 webserver.cc:492] Webserver started at http://127.27.17.194:41415/ using document root <none> and password file <none>
I20260429 20:00:53.417954 4868 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:53.418048 4868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:53.421837 4868 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:53.424093 4883 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.425272 4868 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:00:53.425523 4868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "a4605d3669e84d0bac3adc1c05aeec04"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d7ffb95b5e0afa33c990eaa926ab8ba9"
server_key_iv: "e81affba9383dd95c371c3d89ce02cc2"
server_key_version: "encryptionkey@0"
I20260429 20:00:53.426031 4868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:53.435882 4868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:53.438385 4868 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:53.438599 4868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:53.439208 4868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:53.440096 4868 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:53.440171 4868 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.440243 4868 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:53.440304 4868 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.450307 4868 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:38695
I20260429 20:00:53.450312 4996 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:38695 every 8 connection(s)
I20260429 20:00:53.451584 4868 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:00:53.452248 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4868
I20260429 20:00:53.452785 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:44113
--local_ip_for_outbound_sockets=127.27.17.195
--tserver_master_addrs=127.27.17.254:37961
--webserver_port=35873
--webserver_interface=127.27.17.195
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:41851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.17.195
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 20:00:53.464131 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.453776 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:35159 (local address 127.27.17.254:37961)
0429 20:00:53.453945 (+ 169us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.453950 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.454659 (+ 709us) server_negotiation.cc:408] Connection header received
0429 20:00:53.455940 (+ 1281us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.455943 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.456003 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.456090 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.458378 (+ 2288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.458974 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.460002 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.460172 (+ 170us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.464018 (+ 3846us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:35159: BlockingRecv error: recv got EOF from 127.27.17.194:35159 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260429 20:00:53.464731 5000 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.454049 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:35159)
0429 20:00:53.454504 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.454553 (+ 49us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.455679 (+ 1126us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.456292 (+ 613us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.456305 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.456892 (+ 587us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.458160 (+ 1268us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.458183 (+ 23us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.459112 (+ 929us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.459118 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.459846 (+ 728us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.459859 (+ 13us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.460105 (+ 246us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.460784 (+ 679us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.460817 (+ 33us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.463842 (+ 3025us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":163,"threads_started":1}
W20260429 20:00:53.465173 4997 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:37961 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.17.254:37961: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:53.471243 5000 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.465744 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:47147)
0429 20:00:53.466006 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.466020 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.466109 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.466462 (+ 353us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.466465 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.466620 (+ 155us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.466826 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.466832 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.468007 (+ 1175us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.468011 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.468435 (+ 424us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.468442 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.468550 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.469117 (+ 567us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.469132 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.471131 (+ 1999us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260429 20:00:53.471417 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.465897 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:47147 (local address 127.27.17.254:37961)
0429 20:00:53.466063 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.466070 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.466089 (+ 19us) server_negotiation.cc:408] Connection header received
0429 20:00:53.466215 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.466220 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.466283 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.466383 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.467049 (+ 666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.467870 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.468571 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.468775 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.471291 (+ 2516us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:47147: BlockingRecv error: recv got EOF from 127.27.17.194:47147 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:53.476609 5000 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.471922 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:42227)
0429 20:00:53.472055 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.472070 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.472160 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.472472 (+ 312us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.472474 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.472654 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.472814 (+ 160us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.472819 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.473546 (+ 727us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.473553 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.473909 (+ 356us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.473915 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.474038 (+ 123us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.474578 (+ 540us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.474591 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.476474 (+ 1883us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260429 20:00:53.476858 4997 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260429 20:00:53.476883 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.472000 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:42227 (local address 127.27.17.254:37961)
0429 20:00:53.472129 (+ 129us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.472133 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.472146 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:53.472316 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.472319 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.472363 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.472454 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.472930 (+ 476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.473427 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.474020 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.474220 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.476642 (+ 2422us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:42227: BlockingRecv error: recv got EOF from 127.27.17.194:42227 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260429 20:00:53.562597 4999 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:00:53.562827 4999 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:00:53.562911 4999 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:00:53.566370 4999 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:00:53.566463 4999 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:00:53.566550 4999 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:00:53.571234 4999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:41851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/krb5kdc/oryx_127.27.17.195.keytab
--principal=oryx/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:44113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=35873
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:00:53.572341 4999 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:00:53.573161 4999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:00:53.579859 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.579859 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:00:53.579875 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:00:53.580186 4999 server_base.cc:1061] running on GCE node
I20260429 20:00:53.580871 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:00:53.581450 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:00:53.582653 4999 hybrid_clock.cc:648] HybridClock initialized: now 1777492853582629 us; error 40 us; skew 500 ppm
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492853, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:00:53.585776 4999 init.cc:377] Logged in from keytab as oryx/127.27.17.195@KRBTEST.COM (short username oryx)
I20260429 20:00:53.586993 4999 webserver.cc:492] Webserver started at http://127.27.17.195:35873/ using document root <none> and password file <none>
I20260429 20:00:53.587584 4999 fs_manager.cc:362] Metadata directory not provided
I20260429 20:00:53.587647 4999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:00:53.591491 4999 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260429 20:00:53.593662 5015 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.594825 4999 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:00:53.595032 4999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "58a1b1cb4b8a46f29d1d0865d56db896"
format_stamp: "Formatted at 2026-04-29 20:00:52 on dist-test-slave-01nr"
server_key: "d1f5cb2875f4be9989c836d65bc202c4"
server_key_iv: "e58d45a611366cfaa3e52294b39aa18e"
server_key_version: "encryptionkey@0"
I20260429 20:00:53.595528 4999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:00:53.610531 4999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:00:53.613597 4999 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:00:53.613801 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:00:53.614387 4999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:00:53.615381 4999 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:00:53.615453 4999 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.615525 4999 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:00:53.615573 4999 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:00:53.626192 4999 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:44113
I20260429 20:00:53.626240 5128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:44113 every 8 connection(s)
I20260429 20:00:53.627336 4999 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:00:53.630120 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 4999
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
W20260429 20:00:53.638296 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.629127 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:36901 (local address 127.27.17.254:37961)
0429 20:00:53.629279 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.629284 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.630010 (+ 726us) server_negotiation.cc:408] Connection header received
0429 20:00:53.631131 (+ 1121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.631135 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.631197 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.631294 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.633348 (+ 2054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.633826 (+ 478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.634770 (+ 944us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.634990 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.638168 (+ 3178us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:36901: BlockingRecv error: recv got EOF from 127.27.17.195:36901 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260429 20:00:53.638887 5131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.629354 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:36901)
0429 20:00:53.629791 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.629824 (+ 33us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.630868 (+ 1044us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.631595 (+ 727us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.631609 (+ 14us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.632195 (+ 586us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.633158 (+ 963us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.633172 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.633956 (+ 784us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.633960 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.634617 (+ 657us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.634626 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.634885 (+ 259us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.635535 (+ 650us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.635557 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.638040 (+ 2483us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
W20260429 20:00:53.639379 5129 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:37961 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.17.254:37961: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
I20260429 20:00:53.640383 5133 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.631266 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.27.17.254:37961)
0429 20:00:53.631540 (+ 274us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.631544 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.631557 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:53.631616 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.631619 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.631656 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.631762 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.632679 (+ 917us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.633212 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.633913 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.634126 (+ 213us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.635495 (+ 1369us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:00:53.635521 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:00:53.635536 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:00:53.635567 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:00:53.637669 (+ 2102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:53.638170 (+ 501us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:53.638177 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:53.638183 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:53.638273 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:00:53.638743 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:00:53.638745 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:00:53.638747 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:00:53.639104 (+ 357us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:00:53.639284 (+ 180us) server_negotiation.cc:1036] Waiting for connection context
0429 20:00:53.640010 (+ 726us) server_negotiation.cc:300] Negotiation successful
0429 20:00:53.640252 (+ 242us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":102,"threads_started":1}
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:53.644965 5131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.639909 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:46351)
0429 20:00:53.640033 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.640051 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.640133 (+ 82us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.640438 (+ 305us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.640441 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.640603 (+ 162us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.640789 (+ 186us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.640794 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.641639 (+ 845us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.641643 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.642147 (+ 504us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.642155 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.642246 (+ 91us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.642778 (+ 532us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.642792 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.644815 (+ 2023us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260429 20:00:53.645032 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.640108 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:46351 (local address 127.27.17.254:37961)
0429 20:00:53.640240 (+ 132us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.640244 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.640256 (+ 12us) server_negotiation.cc:408] Connection header received
0429 20:00:53.640288 (+ 32us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.640291 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.640335 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.640420 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.640905 (+ 485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.641529 (+ 624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.642259 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.642464 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.644920 (+ 2456us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:46351: BlockingRecv error: recv got EOF from 127.27.17.195:46351 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:53 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:53.650112 5131 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:53.645607 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:34157)
0429 20:00:53.645752 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:53.645767 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:53.645857 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:53.646099 (+ 242us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:53.646102 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:53.646249 (+ 147us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:53.646434 (+ 185us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.646440 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.647193 (+ 753us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.647198 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:53.647572 (+ 374us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:53.647579 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.647705 (+ 126us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.648230 (+ 525us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:53.648244 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:53.649961 (+ 1717us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260429 20:00:53.650179 4866 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:53.645640 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:34157 (local address 127.27.17.254:37961)
0429 20:00:53.645787 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:53.645793 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:53.645808 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:53.645958 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:53.645961 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:53.646007 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:53.646083 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:53.646556 (+ 473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.647091 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:53.647685 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:53.647885 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:53.650074 (+ 2189us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:34157: BlockingRecv error: recv got EOF from 127.27.17.195:34157 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260429 20:00:53.650380 5129 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Apr 29 20:00:54 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:54 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:54.289969 5134 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:54.284542 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:55033)
0429 20:00:54.284851 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:54.284870 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:54.284999 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:54.285285 (+ 286us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:54.285288 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:54.285516 (+ 228us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:54.285717 (+ 201us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:54.285723 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.286555 (+ 832us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:54.286559 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:54.287031 (+ 472us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:54.287039 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.287167 (+ 128us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:54.287770 (+ 603us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:54.287786 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:54.289830 (+ 2044us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
W20260429 20:00:54.290190 5135 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:54.284722 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:55033 (local address 127.27.17.254:37961)
0429 20:00:54.284993 (+ 271us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:54.284996 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:54.285011 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:54.285078 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:54.285080 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:54.285127 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:54.285208 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:54.285872 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.286418 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:54.287147 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.287365 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:54.289996 (+ 2631us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:55033: BlockingRecv error: recv got EOF from 127.27.17.193:55033 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
Apr 29 20:00:54 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:54 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:54.483260 5136 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:54.477702 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:51599)
0429 20:00:54.477985 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:54.478000 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:54.478106 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:54.478502 (+ 396us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:54.478506 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:54.478745 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:54.478998 (+ 253us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:54.479004 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.479943 (+ 939us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:54.479948 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:54.480365 (+ 417us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:54.480372 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.480508 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:54.481078 (+ 570us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:54.481091 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:54.483129 (+ 2038us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":108,"threads_started":1}
W20260429 20:00:54.483494 5135 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:54.477861 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:51599 (local address 127.27.17.254:37961)
0429 20:00:54.478006 (+ 145us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:54.478009 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:54.478074 (+ 65us) server_negotiation.cc:408] Connection header received
0429 20:00:54.478238 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:54.478242 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:54.478339 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:54.478425 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:54.479160 (+ 735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.479816 (+ 656us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:54.480490 (+ 674us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.480698 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:54.483365 (+ 2667us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:51599: BlockingRecv error: recv got EOF from 127.27.17.194:51599 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 20:00:54 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:54 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:54.656544 5137 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:54.651121 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:38361)
0429 20:00:54.651565 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:54.651582 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:54.651733 (+ 151us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:54.652008 (+ 275us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:54.652011 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:54.652238 (+ 227us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:54.652474 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:54.652481 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.653236 (+ 755us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:54.653240 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:54.653639 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:54.653645 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.653776 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:54.654386 (+ 610us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:54.654399 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:54.656410 (+ 2011us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":100,"threads_started":1}
W20260429 20:00:54.656869 5135 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:54.651245 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:38361 (local address 127.27.17.254:37961)
0429 20:00:54.651535 (+ 290us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:54.651540 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:54.651667 (+ 127us) server_negotiation.cc:408] Connection header received
0429 20:00:54.651837 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:54.651840 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:54.651894 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:54.651984 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:54.652611 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.653119 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:54.653756 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:54.653978 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:54.656631 (+ 2653us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:38361: BlockingRecv error: recv got EOF from 127.27.17.195:38361 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Apr 29 20:00:55 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:55 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:55.302421 5138 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:55.290887 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:57811)
0429 20:00:55.291229 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:55.291244 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:55.291350 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:55.291642 (+ 292us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:55.291645 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:55.291827 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:55.292024 (+ 197us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:55.292029 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.293058 (+ 1029us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:55.293061 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:55.293465 (+ 404us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:55.293472 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.293625 (+ 153us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:55.299889 (+ 6264us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:55.299911 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:55.302215 (+ 2304us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":110,"threads_started":1}
W20260429 20:00:55.302562 5139 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:55.291094 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:57811 (local address 127.27.17.254:37961)
0429 20:00:55.291363 (+ 269us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:55.291366 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:55.291402 (+ 36us) server_negotiation.cc:408] Connection header received
0429 20:00:55.291443 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:55.291446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:55.291493 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:55.291573 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:55.292179 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.292895 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:55.293598 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.293789 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:55.302385 (+ 8596us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:57811: BlockingRecv error: recv got EOF from 127.27.17.193:57811 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
Apr 29 20:00:55 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:55 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:55.490716 5140 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:55.484356 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:50383)
0429 20:00:55.484720 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:55.484744 (+ 24us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:55.484845 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:55.485296 (+ 451us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:55.485300 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:55.485640 (+ 340us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:55.485843 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:55.485850 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.486753 (+ 903us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:55.486757 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:55.487261 (+ 504us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:55.487270 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.487426 (+ 156us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:55.488044 (+ 618us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:55.488059 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:55.490551 (+ 2492us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
W20260429 20:00:55.490820 5139 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:55.484605 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:50383 (local address 127.27.17.254:37961)
0429 20:00:55.484771 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:55.484775 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:55.484811 (+ 36us) server_negotiation.cc:408] Connection header received
0429 20:00:55.484991 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:55.484995 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:55.485046 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:55.485227 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:55.486000 (+ 773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.486609 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:55.487457 (+ 848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.487643 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:55.490707 (+ 3064us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:50383: BlockingRecv error: recv got EOF from 127.27.17.194:50383 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Apr 29 20:00:55 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:55 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:55.663261 5141 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:55.657458 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:40809)
0429 20:00:55.657839 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:55.657857 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:55.658018 (+ 161us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:55.658321 (+ 303us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:55.658324 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:55.658530 (+ 206us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:55.658723 (+ 193us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:55.658728 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.659585 (+ 857us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:55.659589 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:55.660066 (+ 477us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:55.660074 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.660213 (+ 139us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:55.660884 (+ 671us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:55.660898 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:55.663120 (+ 2222us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":140,"threads_started":1}
W20260429 20:00:55.663486 5139 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:55.657673 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:40809 (local address 127.27.17.254:37961)
0429 20:00:55.657841 (+ 168us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:55.657848 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:55.657949 (+ 101us) server_negotiation.cc:408] Connection header received
0429 20:00:55.658125 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:55.658129 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:55.658188 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:55.658282 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:55.658858 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.659430 (+ 572us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:55.660316 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:55.660527 (+ 211us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:55.663344 (+ 2817us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:40809: BlockingRecv error: recv got EOF from 127.27.17.195:40809 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Apr 29 20:00:56 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:56 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:56.309433 5142 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:56.303628 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:57359)
0429 20:00:56.303929 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:56.303943 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:56.304042 (+ 99us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:56.304343 (+ 301us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:56.304346 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:56.304559 (+ 213us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:56.304774 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:56.304780 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.305655 (+ 875us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:56.305659 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:56.306068 (+ 409us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:56.306075 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.306186 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:56.306833 (+ 647us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:56.306847 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:56.309298 (+ 2451us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
W20260429 20:00:56.309662 5143 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:56.303776 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:57359 (local address 127.27.17.254:37961)
0429 20:00:56.304032 (+ 256us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:56.304035 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:56.304050 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:00:56.304118 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:56.304120 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:56.304169 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:56.304298 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:56.304929 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.305507 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:56.306188 (+ 681us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.306365 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:56.309512 (+ 3147us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:57359: BlockingRecv error: recv got EOF from 127.27.17.193:57359 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
Apr 29 20:00:56 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:56 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:56.497036 5144 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:56.491719 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:58565)
0429 20:00:56.492006 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:56.492020 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:56.492123 (+ 103us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:56.492385 (+ 262us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:56.492388 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:56.492582 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:56.492778 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:56.492783 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.493621 (+ 838us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:56.493625 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:56.494012 (+ 387us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:56.494019 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.494127 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:56.494721 (+ 594us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:56.494736 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:56.496896 (+ 2160us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
W20260429 20:00:56.497293 5143 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:56.491855 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:58565 (local address 127.27.17.254:37961)
0429 20:00:56.491999 (+ 144us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:56.492002 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:56.492089 (+ 87us) server_negotiation.cc:408] Connection header received
0429 20:00:56.492203 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:56.492205 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:56.492252 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:56.492358 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:56.492916 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.493475 (+ 559us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:56.494170 (+ 695us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.494406 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:56.497065 (+ 2659us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:58565: BlockingRecv error: recv got EOF from 127.27.17.194:58565 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Apr 29 20:00:56 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:56 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:56.670022 5145 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:56.664160 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:55775)
0429 20:00:56.664465 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:56.664480 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:56.664644 (+ 164us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:56.664990 (+ 346us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:56.664993 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:56.665189 (+ 196us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:56.665425 (+ 236us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:56.665430 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.666286 (+ 856us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:56.666290 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:56.666727 (+ 437us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:56.666735 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.666845 (+ 110us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:56.667495 (+ 650us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:56.667509 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:56.669889 (+ 2380us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260429 20:00:56.670221 5143 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:56.664259 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:55775 (local address 127.27.17.254:37961)
0429 20:00:56.664487 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:56.664491 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:56.664609 (+ 118us) server_negotiation.cc:408] Connection header received
0429 20:00:56.664768 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:56.664771 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:56.664826 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:56.664911 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:56.665632 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.666122 (+ 490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:56.666933 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:56.667153 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:56.670073 (+ 2920us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:55775: BlockingRecv error: recv got EOF from 127.27.17.195:55775 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Apr 29 20:00:57 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:57 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:57.315724 5146 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:57.310292 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:33755)
0429 20:00:57.310703 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:57.310724 (+ 21us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:57.310843 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:57.311203 (+ 360us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:57.311206 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:57.311451 (+ 245us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:57.311654 (+ 203us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:57.311661 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.312481 (+ 820us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:57.312484 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:57.312872 (+ 388us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:57.312878 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.312977 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:57.313558 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:57.313575 (+ 17us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:57.315610 (+ 2035us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":116,"threads_started":1}
W20260429 20:00:57.315953 5147 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:57.310387 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:33755 (local address 127.27.17.254:37961)
0429 20:00:57.310670 (+ 283us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:57.310674 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:57.310804 (+ 130us) server_negotiation.cc:408] Connection header received
0429 20:00:57.310990 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:57.310993 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:57.311044 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:57.311165 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:57.311772 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.312302 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:57.312980 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.313140 (+ 160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:57.315832 (+ 2692us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:33755: BlockingRecv error: recv got EOF from 127.27.17.193:33755 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":103,"threads_started":1}
Apr 29 20:00:57 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:57 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:57.503407 5148 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:57.497888 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:60057)
0429 20:00:57.498203 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:57.498216 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:57.498394 (+ 178us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:57.498696 (+ 302us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:57.498699 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:57.498938 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:57.499129 (+ 191us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:57.499135 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.499887 (+ 752us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:57.499891 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:57.500378 (+ 487us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:57.500386 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.500517 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:57.501160 (+ 643us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:57.501175 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:57.503285 (+ 2110us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":89,"threads_started":1}
W20260429 20:00:57.503707 5147 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:57.497949 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:60057 (local address 127.27.17.254:37961)
0429 20:00:57.498121 (+ 172us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:57.498125 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:57.498331 (+ 206us) server_negotiation.cc:408] Connection header received
0429 20:00:57.498521 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:57.498524 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:57.498570 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:57.498675 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:57.499250 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.499771 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:57.500499 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.500705 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:57.503440 (+ 2735us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:60057: BlockingRecv error: recv got EOF from 127.27.17.194:60057 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Apr 29 20:00:57 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:57 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:57.676172 5149 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:57.671044 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:57443)
0429 20:00:57.671333 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:57.671346 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:57.671446 (+ 100us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:57.671816 (+ 370us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:57.671819 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:57.671999 (+ 180us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:57.672204 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:57.672209 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.672996 (+ 787us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:57.672999 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:57.673421 (+ 422us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:57.673428 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.673559 (+ 131us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:57.674167 (+ 608us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:57.674180 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:57.675999 (+ 1819us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":110,"threads_started":1}
W20260429 20:00:57.676352 5147 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:57.671114 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:57443 (local address 127.27.17.254:37961)
0429 20:00:57.671268 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:57.671272 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:57.671415 (+ 143us) server_negotiation.cc:408] Connection header received
0429 20:00:57.671582 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:57.671585 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:57.671644 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:57.671739 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:57.672353 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.672862 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:57.673548 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:57.673682 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:57.676161 (+ 2479us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:57443: BlockingRecv error: recv got EOF from 127.27.17.195:57443 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Apr 29 20:00:58 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:58 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:58.323328 5150 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:58.316792 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:48681)
0429 20:00:58.317402 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:58.317420 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:58.317566 (+ 146us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:58.318087 (+ 521us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:58.318091 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:58.318377 (+ 286us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:58.318757 (+ 380us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:58.318767 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.319849 (+ 1082us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:58.319852 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:58.320233 (+ 381us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:58.320239 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.320359 (+ 120us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:58.321004 (+ 645us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:58.321019 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:58.323199 (+ 2180us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
W20260429 20:00:58.323575 5151 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:58.316863 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:48681 (local address 127.27.17.254:37961)
0429 20:00:58.317161 (+ 298us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:58.317166 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:58.317539 (+ 373us) server_negotiation.cc:408] Connection header received
0429 20:00:58.317840 (+ 301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:58.317845 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:58.317914 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:58.318008 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:58.319092 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.319701 (+ 609us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:58.320402 (+ 701us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.320611 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:58.323430 (+ 2819us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:48681: BlockingRecv error: recv got EOF from 127.27.17.193:48681 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":83,"threads_started":1}
Apr 29 20:00:58 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:58 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:58.509254 5152 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:58.504298 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:59701)
0429 20:00:58.504658 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:58.504672 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:58.504765 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:58.505044 (+ 279us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:58.505047 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:58.505252 (+ 205us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:58.505458 (+ 206us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:58.505463 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.506215 (+ 752us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:58.506217 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:58.506608 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:58.506614 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.506712 (+ 98us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:58.507337 (+ 625us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:58.507351 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:58.509138 (+ 1787us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":149,"threads_started":1}
W20260429 20:00:58.509461 5151 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:58.504471 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:59701 (local address 127.27.17.254:37961)
0429 20:00:58.504713 (+ 242us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:58.504717 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:58.504730 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:00:58.504855 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:58.504858 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:58.504902 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:58.504975 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:58.505614 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.506115 (+ 501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:58.506717 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.506852 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:58.509276 (+ 2424us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:59701: BlockingRecv error: recv got EOF from 127.27.17.194:59701 (error 108)
Metrics: {"server-negotiator.queue_time_us":157}
Apr 29 20:00:58 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:58 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:58.682601 5153 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:58.677178 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:45293)
0429 20:00:58.677510 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:58.677541 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:58.677647 (+ 106us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:58.677908 (+ 261us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:58.677911 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:58.678095 (+ 184us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:58.678311 (+ 216us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:58.678316 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.679205 (+ 889us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:58.679209 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:58.679645 (+ 436us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:58.679652 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.679763 (+ 111us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:58.680320 (+ 557us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:58.680363 (+ 43us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:58.682453 (+ 2090us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":160,"threads_started":1}
W20260429 20:00:58.682801 5151 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:58.677320 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:45293 (local address 127.27.17.254:37961)
0429 20:00:58.677486 (+ 166us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:58.677491 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:58.677652 (+ 161us) server_negotiation.cc:408] Connection header received
0429 20:00:58.677732 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:58.677735 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:58.677798 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:58.677865 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:58.678506 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.679055 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:58.679777 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:58.679932 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:58.682662 (+ 2730us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:45293: BlockingRecv error: recv got EOF from 127.27.17.195:45293 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 20:00:59 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:59 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:59.329370 5154 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:59.324231 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:49281)
0429 20:00:59.324602 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:59.324617 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:59.324734 (+ 117us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:59.325042 (+ 308us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:59.325044 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:59.325231 (+ 187us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:59.325438 (+ 207us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:59.325444 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.326274 (+ 830us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:59.326277 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:59.326676 (+ 399us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:59.326683 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.326783 (+ 100us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:59.327442 (+ 659us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:59.327457 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:59.329244 (+ 1787us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":98,"threads_started":1}
W20260429 20:00:59.329632 5155 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:59.324387 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:49281 (local address 127.27.17.254:37961)
0429 20:00:59.324642 (+ 255us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:59.324645 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:59.324677 (+ 32us) server_negotiation.cc:408] Connection header received
0429 20:00:59.324865 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:59.324869 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:59.324922 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:59.325008 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:59.325594 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.326155 (+ 561us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:59.326789 (+ 634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.327024 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:59.329458 (+ 2434us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:49281: BlockingRecv error: recv got EOF from 127.27.17.193:49281 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":61,"threads_started":1}
Apr 29 20:00:59 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:59 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:59.515995 5156 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:59.510187 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:42877)
0429 20:00:59.510506 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:59.510523 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:59.510701 (+ 178us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:59.511016 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:59.511020 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:59.511259 (+ 239us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:59.511510 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:59.511517 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.512289 (+ 772us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:59.512295 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:59.512702 (+ 407us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:59.512709 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.512842 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:59.513549 (+ 707us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:59.513587 (+ 38us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:59.515823 (+ 2236us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":119,"threads_started":1}
W20260429 20:00:59.516054 5155 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:59.510272 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:42877 (local address 127.27.17.254:37961)
0429 20:00:59.510484 (+ 212us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:59.510489 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:59.510629 (+ 140us) server_negotiation.cc:408] Connection header received
0429 20:00:59.510815 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:59.510825 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:59.510874 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:59.510993 (+ 119us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:59.511641 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.512172 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:59.512822 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.513023 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:59.515950 (+ 2927us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:42877: BlockingRecv error: recv got EOF from 127.27.17.194:42877 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Apr 29 20:00:59 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:00:59 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:00:59.689581 5157 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:00:59.683843 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:40503)
0429 20:00:59.684263 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0429 20:00:59.684282 (+ 19us) client_negotiation.cc:175] Beginning negotiation
0429 20:00:59.684419 (+ 137us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:00:59.684817 (+ 398us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:00:59.684820 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:00:59.685079 (+ 259us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:00:59.685344 (+ 265us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:59.685351 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.686124 (+ 773us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:00:59.686130 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:00:59.686527 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:00:59.686533 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.686666 (+ 133us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:59.687306 (+ 640us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:00:59.687321 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:00:59.689399 (+ 2078us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":113,"threads_started":1}
W20260429 20:00:59.689653 5155 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:00:59.684014 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:40503 (local address 127.27.17.254:37961)
0429 20:00:59.684167 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 20:00:59.684172 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:00:59.684396 (+ 224us) server_negotiation.cc:408] Connection header received
0429 20:00:59.684595 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:00:59.684600 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:00:59.684687 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:00:59.684774 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:00:59.685482 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.686005 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:00:59.686649 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:00:59.686865 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:00:59.689533 (+ 2668us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:40503: BlockingRecv error: recv got EOF from 127.27.17.195:40503 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Apr 29 20:01:00 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:00 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:00.340366 5158 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:00.330231 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:34875)
0429 20:01:00.330533 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:00.330547 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:00.330690 (+ 143us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:00.331096 (+ 406us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:00.331098 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:00.331286 (+ 188us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:00.331512 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:00.331517 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.332236 (+ 719us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:00.332238 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:00.332639 (+ 401us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:00.332645 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.332739 (+ 94us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:00.338191 (+ 5452us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:00.338209 (+ 18us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:00.340238 (+ 2029us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":97,"threads_started":1}
W20260429 20:01:00.340579 5159 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:00.330347 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:34875 (local address 127.27.17.254:37961)
0429 20:01:00.330590 (+ 243us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:00.330593 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:00.330608 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:01:00.330846 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:00.330850 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:00.330958 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:00.331058 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:00.331628 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.332139 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:00.332742 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.332882 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:00.340430 (+ 7548us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:34875: BlockingRecv error: recv got EOF from 127.27.17.193:34875 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":89,"threads_started":1}
Apr 29 20:01:00 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:00 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:00.522236 5160 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:00.516970 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:55539)
0429 20:01:00.517421 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:00.517437 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:00.517562 (+ 125us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:00.517843 (+ 281us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:00.517846 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:00.518051 (+ 205us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:00.518275 (+ 224us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:00.518285 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.519134 (+ 849us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:00.519137 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:00.519547 (+ 410us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:00.519554 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.519649 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:00.520211 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:00.520224 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:00.522117 (+ 1893us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":85,"threads_started":1}
W20260429 20:01:00.522441 5159 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:00.517128 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:55539 (local address 127.27.17.254:37961)
0429 20:01:00.517280 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:00.517284 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:00.517559 (+ 275us) server_negotiation.cc:408] Connection header received
0429 20:01:00.517639 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:00.517642 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:00.517702 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:00.517811 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:00.518474 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.519031 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:00.519653 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.519801 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:00.522301 (+ 2500us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:55539: BlockingRecv error: recv got EOF from 127.27.17.194:55539 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Apr 29 20:01:00 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:00 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:00.696731 5161 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:00.690509 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:47065)
0429 20:01:00.690833 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:00.690848 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:00.690996 (+ 148us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:00.691615 (+ 619us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:00.691618 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:00.691830 (+ 212us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:00.692025 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:00.692030 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.692826 (+ 796us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:00.692829 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:00.693255 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:00.693263 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.693382 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:00.694130 (+ 748us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:00.694145 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:00.696592 (+ 2447us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":91,"threads_started":1}
W20260429 20:01:00.697069 5159 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:00.690647 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:47065 (local address 127.27.17.254:37961)
0429 20:01:00.690825 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:00.690829 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:00.690962 (+ 133us) server_negotiation.cc:408] Connection header received
0429 20:01:00.691356 (+ 394us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:00.691364 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:00.691433 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:00.691521 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:00.692153 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.692683 (+ 530us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:00.693411 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:00.693743 (+ 332us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:00.696938 (+ 3195us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:47065: BlockingRecv error: recv got EOF from 127.27.17.195:47065 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Apr 29 20:01:01 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:01 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:01.347445 5162 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:01.341401 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:49827)
0429 20:01:01.341713 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:01.341727 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:01.341828 (+ 101us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:01.342180 (+ 352us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:01.342183 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:01.342456 (+ 273us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:01.342679 (+ 223us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:01.342686 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.343563 (+ 877us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:01.343566 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:01.343963 (+ 397us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:01.343969 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.344087 (+ 118us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:01.344879 (+ 792us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:01.344901 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:01.347253 (+ 2352us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":119,"threads_started":1}
W20260429 20:01:01.347544 5163 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:01.341622 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:49827 (local address 127.27.17.254:37961)
0429 20:01:01.341888 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:01.341891 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:01.341904 (+ 13us) server_negotiation.cc:408] Connection header received
0429 20:01:01.341946 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:01.341948 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:01.341995 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:01.342088 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:01.342822 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.343456 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:01.344149 (+ 693us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.344464 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:01.347395 (+ 2931us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:49827: BlockingRecv error: recv got EOF from 127.27.17.193:49827 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
Apr 29 20:01:01 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:01 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:01.528836 5164 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:01.523233 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:46061)
0429 20:01:01.523612 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:01.523626 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:01.523735 (+ 109us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:01.523991 (+ 256us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:01.523993 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:01.524184 (+ 191us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:01.524381 (+ 197us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:01.524387 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.525314 (+ 927us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:01.525319 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:01.525736 (+ 417us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:01.525745 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.525848 (+ 103us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:01.526447 (+ 599us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:01.526461 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:01.528674 (+ 2213us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":141,"threads_started":1}
W20260429 20:01:01.529026 5163 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:01.523368 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:46061 (local address 127.27.17.254:37961)
0429 20:01:01.523553 (+ 185us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:01.523557 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:01.523706 (+ 149us) server_negotiation.cc:408] Connection header received
0429 20:01:01.523821 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:01.523824 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:01.523872 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:01.523952 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:01.524497 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.525177 (+ 680us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:01.525883 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.526038 (+ 155us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:01.528866 (+ 2828us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:46061: BlockingRecv error: recv got EOF from 127.27.17.194:46061 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Apr 29 20:01:01 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:01 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:01.703372 5165 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:01.697746 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:41091)
0429 20:01:01.698104 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:01.698122 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:01.698286 (+ 164us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:01.698618 (+ 332us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:01.698621 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:01.698834 (+ 213us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:01.699088 (+ 254us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:01.699095 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.700084 (+ 989us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:01.700088 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:01.700588 (+ 500us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:01.700600 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.700721 (+ 121us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:01.701308 (+ 587us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:01.701336 (+ 28us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:01.703240 (+ 1904us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
W20260429 20:01:01.703645 5163 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:01.697914 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:41091 (local address 127.27.17.254:37961)
0429 20:01:01.698105 (+ 191us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:01.698111 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:01.698218 (+ 107us) server_negotiation.cc:408] Connection header received
0429 20:01:01.698428 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:01.698431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:01.698491 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:01.698600 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:01.699284 (+ 684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.699950 (+ 666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:01.700789 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:01.701031 (+ 242us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:01.703519 (+ 2488us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:41091: BlockingRecv error: recv got EOF from 127.27.17.195:41091 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Apr 29 20:01:02 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:02 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:02.354053 5166 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:02.348420 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:57393)
0429 20:01:02.348805 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:02.348820 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:02.348917 (+ 97us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:02.349199 (+ 282us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:02.349201 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:02.349395 (+ 194us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:02.349607 (+ 212us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:02.349613 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.350430 (+ 817us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:02.350433 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:02.350828 (+ 395us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:02.350834 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.350956 (+ 122us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:02.351642 (+ 686us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:02.351657 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:02.353926 (+ 2269us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":129,"threads_started":1}
W20260429 20:01:02.354285 5167 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:02.348578 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:57393 (local address 127.27.17.254:37961)
0429 20:01:02.348851 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:02.348855 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:02.348890 (+ 35us) server_negotiation.cc:408] Connection header received
0429 20:01:02.349024 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:02.349027 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:02.349077 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:02.349155 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:02.349754 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.350286 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:02.351003 (+ 717us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.351198 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:02.354144 (+ 2946us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:57393: BlockingRecv error: recv got EOF from 127.27.17.193:57393 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":66,"threads_started":1}
Apr 29 20:01:02 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:02 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:02.535372 5168 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:02.529814 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:59187)
0429 20:01:02.530225 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:02.530243 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:02.530410 (+ 167us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:02.530725 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:02.530728 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:02.530975 (+ 247us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:02.531220 (+ 245us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:02.531227 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.531971 (+ 744us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:02.531976 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:02.532362 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:02.532368 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.532504 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:02.533115 (+ 611us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:02.533130 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:02.535182 (+ 2052us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":96,"threads_started":1}
W20260429 20:01:02.535430 5167 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:02.529982 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:59187 (local address 127.27.17.254:37961)
0429 20:01:02.530170 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:02.530176 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:02.530336 (+ 160us) server_negotiation.cc:408] Connection header received
0429 20:01:02.530535 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:02.530543 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:02.530609 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:02.530697 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:02.531343 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.531855 (+ 512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:02.532487 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.532714 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:02.535319 (+ 2605us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:59187: BlockingRecv error: recv got EOF from 127.27.17.194:59187 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Apr 29 20:01:02 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:02 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:02.711112 5169 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:02.704643 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:57567)
0429 20:01:02.704996 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:02.705010 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:02.705169 (+ 159us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:02.705515 (+ 346us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:02.705519 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:02.705769 (+ 250us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:02.706055 (+ 286us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:02.706071 (+ 16us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.707160 (+ 1089us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:02.707164 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:02.707591 (+ 427us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:02.707598 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.707733 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:02.708614 (+ 881us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:02.708630 (+ 16us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:02.710971 (+ 2341us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":124,"threads_started":1}
W20260429 20:01:02.711288 5167 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:02.704850 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:57567 (local address 127.27.17.254:37961)
0429 20:01:02.705004 (+ 154us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:02.705009 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:02.705188 (+ 179us) server_negotiation.cc:408] Connection header received
0429 20:01:02.705263 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:02.705268 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:02.705336 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:02.705418 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:02.706224 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.706820 (+ 596us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:02.707768 (+ 948us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:02.707925 (+ 157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:02.711153 (+ 3228us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:57567: BlockingRecv error: recv got EOF from 127.27.17.195:57567 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Apr 29 20:01:03 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:03 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:03.360085 5170 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:03.355043 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:58359)
0429 20:01:03.355339 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:03.355356 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:03.355449 (+ 93us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:03.355715 (+ 266us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:03.355717 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:03.355896 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:03.356086 (+ 190us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:03.356091 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.356848 (+ 757us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:03.356852 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:03.357229 (+ 377us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:03.357234 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.357329 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:03.357902 (+ 573us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:03.357916 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:03.359963 (+ 2047us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
W20260429 20:01:03.360283 5171 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:03.355082 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:58359 (local address 127.27.17.254:37961)
0429 20:01:03.355398 (+ 316us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:03.355401 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:03.355416 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:01:03.355547 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:03.355549 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:03.355598 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:03.355678 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:03.356187 (+ 509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.356734 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:03.357378 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.357536 (+ 158us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:03.360135 (+ 2599us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:58359: BlockingRecv error: recv got EOF from 127.27.17.193:58359 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
Apr 29 20:01:03 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:03 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:03.541057 5172 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:03.536230 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:48983)
0429 20:01:03.536614 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:03.536627 (+ 13us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:03.536722 (+ 95us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:03.536940 (+ 218us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:03.536942 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:03.537123 (+ 181us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:03.537342 (+ 219us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:03.537347 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.538041 (+ 694us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:03.538043 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:03.538443 (+ 400us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:03.538449 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.538538 (+ 89us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:03.539084 (+ 546us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:03.539097 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:03.540938 (+ 1841us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":153,"threads_started":1}
W20260429 20:01:03.541278 5171 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:03.536263 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:48983 (local address 127.27.17.254:37961)
0429 20:01:03.536438 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:03.536442 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:03.536701 (+ 259us) server_negotiation.cc:408] Connection header received
0429 20:01:03.536788 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:03.536790 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:03.536837 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:03.536903 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:03.537443 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.537947 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:03.538545 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.538698 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:03.541134 (+ 2436us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:48983: BlockingRecv error: recv got EOF from 127.27.17.194:48983 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 20:01:03 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:03 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:03.717984 5173 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:03.712117 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:46757)
0429 20:01:03.712555 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:03.712572 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:03.712732 (+ 160us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:03.713015 (+ 283us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:03.713018 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:03.713239 (+ 221us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:03.713435 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:03.713441 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.714419 (+ 978us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:03.714426 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:03.714847 (+ 421us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:03.714856 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.715075 (+ 219us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:03.715690 (+ 615us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:03.715704 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:03.717798 (+ 2094us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":199,"threads_started":1}
W20260429 20:01:03.718045 5171 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:03.712342 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:46757 (local address 127.27.17.254:37961)
0429 20:01:03.712570 (+ 228us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:03.712576 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:03.712671 (+ 95us) server_negotiation.cc:408] Connection header received
0429 20:01:03.712838 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:03.712841 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:03.712901 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:03.712992 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:03.713585 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.714186 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:03.715068 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:03.715283 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:03.717936 (+ 2653us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:46757: BlockingRecv error: recv got EOF from 127.27.17.195:46757 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Apr 29 20:01:04 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:04 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:04.366456 5174 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:04.361103 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:50915)
0429 20:01:04.361379 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:04.361403 (+ 24us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:04.361501 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:04.361800 (+ 299us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:04.361803 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:04.361995 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:04.362204 (+ 209us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:04.362209 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.363056 (+ 847us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:04.363058 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:04.363449 (+ 391us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:04.363456 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.363553 (+ 97us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:04.364134 (+ 581us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:04.364148 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:04.366165 (+ 2017us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
W20260429 20:01:04.366585 5175 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:04.361212 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:50915 (local address 127.27.17.254:37961)
0429 20:01:04.361561 (+ 349us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:04.361564 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:04.361579 (+ 15us) server_negotiation.cc:408] Connection header received
0429 20:01:04.361622 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:04.361624 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:04.361672 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:04.361768 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:04.362369 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.362947 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:04.363580 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.363727 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:04.366431 (+ 2704us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:50915: BlockingRecv error: recv got EOF from 127.27.17.193:50915 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":164,"threads_started":1}
Apr 29 20:01:04 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:04 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:04.547629 5176 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:04.542098 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:56703)
0429 20:01:04.542402 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:04.542416 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:04.542529 (+ 113us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:04.542825 (+ 296us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:04.542828 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:04.543044 (+ 216us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:04.543270 (+ 226us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:04.543275 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.544100 (+ 825us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:04.544104 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:04.544540 (+ 436us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:04.544547 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.544656 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:04.545271 (+ 615us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:04.545285 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:04.547499 (+ 2214us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
W20260429 20:01:04.547904 5175 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:04.542117 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:56703 (local address 127.27.17.254:37961)
0429 20:01:04.542316 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:04.542320 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:04.542532 (+ 212us) server_negotiation.cc:408] Connection header received
0429 20:01:04.542602 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:04.542605 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:04.542655 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:04.542728 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:04.543413 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.543978 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:04.544676 (+ 698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.544847 (+ 171us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:04.547785 (+ 2938us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:56703: BlockingRecv error: recv got EOF from 127.27.17.194:56703 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Apr 29 20:01:04 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:04 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:04.724323 5177 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:04.719071 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:44905)
0429 20:01:04.719395 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:04.719410 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:04.719529 (+ 119us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:04.719814 (+ 285us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:04.719817 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:04.719999 (+ 182us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:04.720195 (+ 196us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:04.720200 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.720968 (+ 768us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:04.720971 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:04.721396 (+ 425us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:04.721402 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.721497 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:04.722059 (+ 562us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:04.722072 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:04.724195 (+ 2123us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":121,"threads_started":1}
W20260429 20:01:04.724670 5175 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:04.719118 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:44905 (local address 127.27.17.254:37961)
0429 20:01:04.719298 (+ 180us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:04.719302 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:04.719524 (+ 222us) server_negotiation.cc:408] Connection header received
0429 20:01:04.719611 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:04.719614 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:04.719671 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:04.719749 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:04.720337 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.720856 (+ 519us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:04.721520 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:04.721740 (+ 220us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:04.724515 (+ 2775us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:44905: BlockingRecv error: recv got EOF from 127.27.17.195:44905 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Apr 29 20:01:05 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:05 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:05.380023 5178 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:05.367525 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:39725)
0429 20:01:05.367987 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:05.368004 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:05.368127 (+ 123us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:05.368474 (+ 347us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:05.368478 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:05.368737 (+ 259us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:05.369025 (+ 288us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:05.369032 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.370031 (+ 999us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:05.370036 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:05.370841 (+ 805us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:05.370855 (+ 14us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.371024 (+ 169us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:05.377311 (+ 6287us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:05.377337 (+ 26us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:05.379884 (+ 2547us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":122,"threads_started":1}
W20260429 20:01:05.380213 5179 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:05.367663 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:39725 (local address 127.27.17.254:37961)
0429 20:01:05.367915 (+ 252us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:05.367921 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:05.368104 (+ 183us) server_negotiation.cc:408] Connection header received
0429 20:01:05.368267 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:05.368274 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:05.368346 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:05.368433 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:05.369163 (+ 730us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.369863 (+ 700us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:05.371007 (+ 1144us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.371273 (+ 266us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:05.380074 (+ 8801us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:39725: BlockingRecv error: recv got EOF from 127.27.17.193:39725 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
Apr 29 20:01:05 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:05 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:05.556376 5180 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:05.549114 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:42003)
0429 20:01:05.549716 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:05.549763 (+ 47us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:05.549990 (+ 227us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:05.550599 (+ 609us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:05.550602 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:05.551157 (+ 555us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:05.551408 (+ 251us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:05.551416 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.552378 (+ 962us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:05.552382 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:05.552808 (+ 426us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:05.552816 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.553045 (+ 229us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:05.553677 (+ 632us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:05.553697 (+ 20us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:05.556173 (+ 2476us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":168,"threads_started":1}
W20260429 20:01:05.556859 5179 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:05.549248 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:42003 (local address 127.27.17.254:37961)
0429 20:01:05.549444 (+ 196us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:05.549451 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:05.549951 (+ 500us) server_negotiation.cc:408] Connection header received
0429 20:01:05.550201 (+ 250us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:05.550205 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:05.550282 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:05.550376 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:05.551676 (+ 1300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.552191 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:05.553029 (+ 838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.553195 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:05.556727 (+ 3532us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:42003: BlockingRecv error: recv got EOF from 127.27.17.194:42003 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Apr 29 20:01:05 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:05 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:05.731952 5181 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:05.725264 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:58759)
0429 20:01:05.725703 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:05.725717 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:05.725815 (+ 98us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:05.726105 (+ 290us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:05.726108 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:05.726305 (+ 197us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:05.726809 (+ 504us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:05.726819 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.727906 (+ 1087us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:05.727910 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:05.728381 (+ 471us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:05.728389 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.728513 (+ 124us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:05.729098 (+ 585us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:05.729113 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:05.731766 (+ 2653us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":76,"threads_started":1}
W20260429 20:01:05.732033 5179 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:05.725677 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:58759 (local address 127.27.17.254:37961)
0429 20:01:05.725824 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:05.725828 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:05.725868 (+ 40us) server_negotiation.cc:408] Connection header received
0429 20:01:05.725906 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:05.725915 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:05.725966 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:05.726030 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:05.727169 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.727734 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:05.728544 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:05.728707 (+ 163us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:05.731917 (+ 3210us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:58759: BlockingRecv error: recv got EOF from 127.27.17.195:58759 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Apr 29 20:01:06 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:06 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:06.387588 5182 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:06.381222 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:46455)
0429 20:01:06.381569 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:06.381585 (+ 16us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:06.381692 (+ 107us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:06.381988 (+ 296us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:06.381991 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:06.382183 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:06.382411 (+ 228us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:06.382416 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.383363 (+ 947us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:06.383375 (+ 12us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:06.383799 (+ 424us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:06.383806 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.383919 (+ 113us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:06.384784 (+ 865us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:06.384799 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:06.387319 (+ 2520us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":93,"threads_started":1}
W20260429 20:01:06.387727 5183 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:06.381338 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:46455 (local address 127.27.17.254:37961)
0429 20:01:06.381689 (+ 351us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:06.381693 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:06.381711 (+ 18us) server_negotiation.cc:408] Connection header received
0429 20:01:06.381781 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:06.381784 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:06.381832 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:06.381912 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:06.382654 (+ 742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.383216 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:06.383975 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.384199 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:06.387583 (+ 3384us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:46455: BlockingRecv error: recv got EOF from 127.27.17.193:46455 (error 108)
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":177,"threads_started":1}
Apr 29 20:01:06 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:06 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:06.563244 5184 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:06.557365 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:48741)
0429 20:01:06.557809 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:06.557829 (+ 20us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:06.557998 (+ 169us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:06.558286 (+ 288us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:06.558289 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:06.558535 (+ 246us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:06.558990 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:06.558997 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.559815 (+ 818us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:06.559822 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:06.560234 (+ 412us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:06.560241 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.560386 (+ 145us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:06.560912 (+ 526us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:06.560926 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:06.563124 (+ 2198us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":150,"threads_started":1}
W20260429 20:01:06.563431 5183 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:06.557622 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:48741 (local address 127.27.17.254:37961)
0429 20:01:06.557811 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:06.557817 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:06.557927 (+ 110us) server_negotiation.cc:408] Connection header received
0429 20:01:06.558111 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:06.558115 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:06.558170 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:06.558262 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:06.559158 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.559693 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:06.560364 (+ 671us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.560612 (+ 248us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:06.563286 (+ 2674us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:48741: BlockingRecv error: recv got EOF from 127.27.17.194:48741 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Apr 29 20:01:06 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:06 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:06.739821 5185 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:06.733125 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:44821)
0429 20:01:06.733611 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:06.733628 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:06.733895 (+ 267us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:06.734305 (+ 410us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:06.734309 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:06.734686 (+ 377us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:06.735143 (+ 457us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:06.735153 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.736082 (+ 929us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:06.736089 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:06.736581 (+ 492us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:06.736593 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.736818 (+ 225us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:06.737494 (+ 676us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:06.737516 (+ 22us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:06.739691 (+ 2175us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":102,"threads_started":1}
W20260429 20:01:06.740149 5183 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:06.733159 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:44821 (local address 127.27.17.254:37961)
0429 20:01:06.733536 (+ 377us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:06.733541 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:06.733705 (+ 164us) server_negotiation.cc:408] Connection header received
0429 20:01:06.734080 (+ 375us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:06.734084 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:06.734150 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:06.734280 (+ 130us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:06.735339 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.735925 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:06.736789 (+ 864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:06.737010 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:06.739967 (+ 2957us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:44821: BlockingRecv error: recv got EOF from 127.27.17.195:44821 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Apr 29 20:01:07 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:07 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:07.394032 5186 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:07.388536 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:35859)
0429 20:01:07.388875 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:07.388890 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:07.388994 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:07.389329 (+ 335us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:07.389331 (+ 2us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:07.389523 (+ 192us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:07.389718 (+ 195us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:07.389723 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.390583 (+ 860us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:07.390585 (+ 2us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:07.391040 (+ 455us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:07.391048 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.391143 (+ 95us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:07.391776 (+ 633us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:07.391790 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:07.393915 (+ 2125us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":94,"threads_started":1}
W20260429 20:01:07.394186 5187 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:07.388687 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:35859 (local address 127.27.17.254:37961)
0429 20:01:07.388982 (+ 295us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:07.388985 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:07.389001 (+ 16us) server_negotiation.cc:408] Connection header received
0429 20:01:07.389151 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:07.389154 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:07.389208 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:07.389292 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:07.389836 (+ 544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.390480 (+ 644us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:07.391150 (+ 670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.391351 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:07.394052 (+ 2701us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:35859: BlockingRecv error: recv got EOF from 127.27.17.193:35859 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":90,"threads_started":1}
Apr 29 20:01:07 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:07 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:07.570065 5188 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:07.564412 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:47165)
0429 20:01:07.564721 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:07.564735 (+ 14us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:07.564864 (+ 129us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:07.565205 (+ 341us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:07.565209 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:07.565479 (+ 270us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:07.565684 (+ 205us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:07.565690 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.566543 (+ 853us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:07.566548 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:07.567102 (+ 554us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:07.567112 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.567231 (+ 119us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:07.567835 (+ 604us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:07.567850 (+ 15us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:07.569884 (+ 2034us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":114,"threads_started":1}
W20260429 20:01:07.570153 5187 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:07.564337 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:47165 (local address 127.27.17.254:37961)
0429 20:01:07.564490 (+ 153us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:07.564494 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:07.564817 (+ 323us) server_negotiation.cc:408] Connection header received
0429 20:01:07.564984 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:07.564988 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:07.565051 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:07.565128 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:07.565841 (+ 713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.566391 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:07.567244 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.567456 (+ 212us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:07.570033 (+ 2577us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:47165: BlockingRecv error: recv got EOF from 127.27.17.194:47165 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Apr 29 20:01:07 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:07 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.195@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:07.746624 5189 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:07.740729 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.195:43569)
0429 20:01:07.741115 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:07.741132 (+ 17us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:07.741292 (+ 160us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:07.741584 (+ 292us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:07.741587 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:07.741795 (+ 208us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:07.741997 (+ 202us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:07.742003 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.742832 (+ 829us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:07.742838 (+ 6us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:07.743386 (+ 548us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:07.743394 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.743529 (+ 135us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:07.744119 (+ 590us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:07.744133 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:07.746333 (+ 2200us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":139,"threads_started":1}
W20260429 20:01:07.746666 5187 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:07.740927 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:43569 (local address 127.27.17.254:37961)
0429 20:01:07.741115 (+ 188us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:07.741121 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:07.741222 (+ 101us) server_negotiation.cc:408] Connection header received
0429 20:01:07.741416 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:07.741419 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:07.741468 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:07.741558 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:07.742121 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.742719 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:07.743507 (+ 788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:07.743700 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:07.746469 (+ 2769us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.195:43569: BlockingRecv error: recv got EOF from 127.27.17.195:43569 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Apr 29 20:01:08 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:08 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.193@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:08.400487 5190 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:08.395041 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.193:42901)
0429 20:01:08.395350 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:08.395365 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:08.395469 (+ 104us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:08.395784 (+ 315us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:08.395787 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:08.395966 (+ 179us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:08.396158 (+ 192us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:08.396163 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.396926 (+ 763us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:08.396930 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:08.397446 (+ 516us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:08.397454 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.397563 (+ 109us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:08.398148 (+ 585us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:08.398161 (+ 13us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:08.400343 (+ 2182us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
W20260429 20:01:08.400662 5191 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:08.395128 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:42901 (local address 127.27.17.254:37961)
0429 20:01:08.395394 (+ 266us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:08.395397 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:08.395413 (+ 16us) server_negotiation.cc:408] Connection header received
0429 20:01:08.395597 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:08.395600 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:08.395650 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:08.395735 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:08.396267 (+ 532us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.396796 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:08.397608 (+ 812us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.397799 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:08.400497 (+ 2698us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.193:42901: BlockingRecv error: recv got EOF from 127.27.17.193:42901 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":75,"threads_started":1}
Apr 29 20:01:08 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
Apr 29 20:01:08 dist-test-slave-01nr krb5kdc[4124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.17.194@KRBTEST.COM for oryx/127.27.17.254@KRBTEST.COM, Server not found in Kerberos database
I20260429 20:01:08.576458 5192 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:08.571252 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:37961 (local address 127.27.17.194:39667)
0429 20:01:08.571590 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:08.571605 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:08.571707 (+ 102us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:08.572001 (+ 294us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:08.572005 (+ 4us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:08.572192 (+ 187us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:08.572392 (+ 200us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:08.572397 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.573125 (+ 728us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:08.573128 (+ 3us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:08.573514 (+ 386us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:08.573520 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.573628 (+ 108us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:08.574258 (+ 630us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:08.574272 (+ 14us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:08.576328 (+ 2056us) negotiation.cc:326] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.17.254:37961: Server oryx/127.27.17.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
W20260429 20:01:08.576669 5191 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:08.571435 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:39667 (local address 127.27.17.254:37961)
0429 20:01:08.571581 (+ 146us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:08.571585 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:08.571710 (+ 125us) server_negotiation.cc:408] Connection header received
0429 20:01:08.571786 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:08.571789 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:08.571840 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:08.571926 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:08.572514 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.573009 (+ 495us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:08.573647 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.573890 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:08.576524 (+ 2634us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.17.194:39667: BlockingRecv error: recv got EOF from 127.27.17.194:39667 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260429 20:01:08.634616 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4716
I20260429 20:01:08.641131 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4868
I20260429 20:01:08.647214 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4999
I20260429 20:01:08.653316 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 4646
2026-04-29T20:01:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (19024 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-04-29T20:01:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:01:08Z Disabled control of system clock
I20260429 20:01:08.685441 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45601
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:45601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260429 20:01:08.800463 5196 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:08.800732 5196 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:08.800777 5196 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:08.804494 5196 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:01:08.804579 5196 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:08.804597 5196 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:08.804618 5196 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:01:08.804637 5196 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:01:08.808900 5196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:45601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:08.810060 5196 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:08.811232 5196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:08.818027 5201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:08.818043 5202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:08.818017 5196 server_base.cc:1061] running on GCE node
W20260429 20:01:08.817991 5204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:08.819094 5196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:08.820125 5196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:08.821339 5196 hybrid_clock.cc:648] HybridClock initialized: now 1777492868821312 us; error 48 us; skew 500 ppm
I20260429 20:01:08.823536 5196 webserver.cc:492] Webserver started at http://127.27.17.254:40543/ using document root <none> and password file <none>
I20260429 20:01:08.824143 5196 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:08.824236 5196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:08.824478 5196 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:08.826275 5196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "fc736a81c3d844e8bdd37002df9a2005"
format_stamp: "Formatted at 2026-04-29 20:01:08 on dist-test-slave-01nr"
server_key: "eb877388f81ded427986de640f0e8b7e"
server_key_iv: "f2295ef29458ccc3e3c3037a90543c46"
server_key_version: "encryptionkey@0"
I20260429 20:01:08.826807 5196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "fc736a81c3d844e8bdd37002df9a2005"
format_stamp: "Formatted at 2026-04-29 20:01:08 on dist-test-slave-01nr"
server_key: "eb877388f81ded427986de640f0e8b7e"
server_key_iv: "f2295ef29458ccc3e3c3037a90543c46"
server_key_version: "encryptionkey@0"
I20260429 20:01:08.830457 5196 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:01:08.833189 5210 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:08.834462 5196 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:01:08.834645 5196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "fc736a81c3d844e8bdd37002df9a2005"
format_stamp: "Formatted at 2026-04-29 20:01:08 on dist-test-slave-01nr"
server_key: "eb877388f81ded427986de640f0e8b7e"
server_key_iv: "f2295ef29458ccc3e3c3037a90543c46"
server_key_version: "encryptionkey@0"
I20260429 20:01:08.834776 5196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:08.857641 5196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:08.858428 5196 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:08.858664 5196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:08.867825 5196 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:45601
I20260429 20:01:08.867944 5262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:45601 every 8 connection(s)
I20260429 20:01:08.869016 5196 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:01:08.872277 5263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:01:08.873034 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5196
I20260429 20:01:08.873195 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:01:08.873533 27719 external_mini_cluster.cc:1468] Setting key c1ad59a2d237c76853acf44e2524a154
I20260429 20:01:08.879920 5263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005: Bootstrap starting.
I20260429 20:01:08.882412 5266 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:08.875022 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52852 (local address 127.27.17.254:45601)
0429 20:01:08.875641 (+ 619us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:08.875650 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:08.875685 (+ 35us) server_negotiation.cc:408] Connection header received
0429 20:01:08.876296 (+ 611us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:08.876315 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:08.876667 (+ 352us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:08.876949 (+ 282us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:08.877637 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.878453 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:08.879546 (+ 1093us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:08.879807 (+ 261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:08.880387 (+ 580us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:08.880452 (+ 65us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:08.880479 (+ 27us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:08.880966 (+ 487us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:08.881018 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:08.881035 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:08.881154 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:08.881459 (+ 305us) server_negotiation.cc:300] Negotiation successful
0429 20:01:08.881625 (+ 166us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":170,"threads_started":1}
I20260429 20:01:08.882735 5263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005: Neither blocks nor log segments found. Creating new log.
I20260429 20:01:08.883677 5263 log.cc:826] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005: Log is configured to *not* fsync() on all Append() calls
I20260429 20:01:08.885747 5263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005: No bootstrap required, opened a new log
I20260429 20:01:08.888736 5263 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc736a81c3d844e8bdd37002df9a2005" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45601 } }
I20260429 20:01:08.888996 5263 raft_consensus.cc:385] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:01:08.889070 5263 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc736a81c3d844e8bdd37002df9a2005, State: Initialized, Role: FOLLOWER
I20260429 20:01:08.889650 5263 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc736a81c3d844e8bdd37002df9a2005" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45601 } }
I20260429 20:01:08.889864 5263 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:01:08.889954 5263 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:01:08.890080 5263 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:01:08.891222 5263 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc736a81c3d844e8bdd37002df9a2005" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45601 } }
I20260429 20:01:08.891808 5263 leader_election.cc:304] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc736a81c3d844e8bdd37002df9a2005; no voters:
I20260429 20:01:08.892320 5263 leader_election.cc:290] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:01:08.892450 5268 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:01:08.892756 5268 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [term 1 LEADER]: Becoming Leader. State: Replica: fc736a81c3d844e8bdd37002df9a2005, State: Running, Role: LEADER
I20260429 20:01:08.893155 5268 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc736a81c3d844e8bdd37002df9a2005" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45601 } }
I20260429 20:01:08.893946 5263 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:01:08.895349 5270 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc736a81c3d844e8bdd37002df9a2005. Latest consensus state: current_term: 1 leader_uuid: "fc736a81c3d844e8bdd37002df9a2005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc736a81c3d844e8bdd37002df9a2005" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45601 } } }
I20260429 20:01:08.895376 5269 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc736a81c3d844e8bdd37002df9a2005" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc736a81c3d844e8bdd37002df9a2005" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45601 } } }
I20260429 20:01:08.895515 5269 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [sys.catalog]: This master's current role is: LEADER
I20260429 20:01:08.895514 5270 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005 [sys.catalog]: This master's current role is: LEADER
W20260429 20:01:08.899416 5283 catalog_manager.cc:1568] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:01:08.899540 5283 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:01:08.899616 5277 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:01:08.900540 5277 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:01:08.906533 5277 catalog_manager.cc:1357] Generated new cluster ID: 9853943f6a184d9b83ea65680dfc9786
I20260429 20:01:08.906738 5277 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:01:08.923053 5277 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:01:08.923983 5277 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:01:08.930975 5277 catalog_manager.cc:6044] T 00000000000000000000000000000000 P fc736a81c3d844e8bdd37002df9a2005: Generated new TSK 0
I20260429 20:01:08.931655 5277 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:01:08.937353 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45601
--builtin_ntp_servers=127.27.17.212:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260429 20:01:09.052204 5287 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:09.052481 5287 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:09.052532 5287 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:09.056303 5287 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:09.056380 5287 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:09.056454 5287 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:01:09.060777 5287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:09.061995 5287 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:09.063277 5287 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:09.070490 5292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:09.070561 5287 server_base.cc:1061] running on GCE node
W20260429 20:01:09.070479 5293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.070479 5295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:09.071292 5287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:09.071943 5287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:09.073170 5287 hybrid_clock.cc:648] HybridClock initialized: now 1777492869073141 us; error 43 us; skew 500 ppm
I20260429 20:01:09.075786 5287 webserver.cc:492] Webserver started at http://127.27.17.193:35377/ using document root <none> and password file <none>
I20260429 20:01:09.076402 5287 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:09.076510 5287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:09.076746 5287 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:09.078773 5287 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "1b1f370e7a8f4208b8fe6e625910472b"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "c0fa0e88c384ed7f51706ab6c726b9eb"
server_key_iv: "8e219e9111dd4a83fffb74e0cd6d2cb4"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.079427 5287 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1f370e7a8f4208b8fe6e625910472b"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "c0fa0e88c384ed7f51706ab6c726b9eb"
server_key_iv: "8e219e9111dd4a83fffb74e0cd6d2cb4"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.083405 5287 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260429 20:01:09.086236 5301 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.087463 5287 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:01:09.087636 5287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "1b1f370e7a8f4208b8fe6e625910472b"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "c0fa0e88c384ed7f51706ab6c726b9eb"
server_key_iv: "8e219e9111dd4a83fffb74e0cd6d2cb4"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.087770 5287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:09.132067 5287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:09.132869 5287 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:09.133124 5287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:09.133792 5287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:09.135038 5287 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:09.135115 5287 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.135190 5287 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:09.135248 5287 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.145722 5287 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:34317
I20260429 20:01:09.145790 5414 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:34317 every 8 connection(s)
I20260429 20:01:09.146838 5287 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:01:09.155354 5266 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.148866 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:57773 (local address 127.27.17.254:45601)
0429 20:01:09.149033 (+ 167us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:09.149038 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:09.149942 (+ 904us) server_negotiation.cc:408] Connection header received
0429 20:01:09.150783 (+ 841us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:09.150787 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:09.150848 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:09.151015 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:09.151917 (+ 902us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.152532 (+ 615us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:09.153229 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.153407 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:09.154732 (+ 1325us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:09.154754 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:09.154756 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:09.154770 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:09.154792 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:09.154801 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:09.154874 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:09.155087 (+ 213us) server_negotiation.cc:300] Negotiation successful
0429 20:01:09.155167 (+ 80us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260429 20:01:09.155368 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5287
I20260429 20:01:09.155488 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:01:09.155783 27719 external_mini_cluster.cc:1468] Setting key ead024a2e9aec7557b5a409ced0c93c1
I20260429 20:01:09.156002 5417 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.149203 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45601 (local address 127.27.17.193:57773)
0429 20:01:09.149798 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:09.149830 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:09.150580 (+ 750us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:09.151180 (+ 600us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:09.151188 (+ 8us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:09.151247 (+ 59us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:09.151759 (+ 512us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:09.151770 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.152677 (+ 907us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:09.152681 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:09.153083 (+ 402us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:09.153089 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.153330 (+ 241us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:09.154485 (+ 1155us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:01:09.154487 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:09.154514 (+ 27us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:01:09.154573 (+ 59us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:09.154890 (+ 317us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:09.154914 (+ 24us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:09.154918 (+ 4us) client_negotiation.cc:770] Sending connection context
0429 20:01:09.155034 (+ 116us) client_negotiation.cc:241] Negotiation successful
0429 20:01:09.155184 (+ 150us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260429 20:01:09.157447 5415 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45601
I20260429 20:01:09.157694 5415 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:09.158049 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45601
--builtin_ntp_servers=127.27.17.212:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260429 20:01:09.158286 5415 heartbeater.cc:507] Master 127.27.17.254:45601 requested a full tablet report, sending...
I20260429 20:01:09.159916 5227 ts_manager.cc:194] Registered new tserver with Master: 1b1f370e7a8f4208b8fe6e625910472b (127.27.17.193:34317)
I20260429 20:01:09.161569 5227 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.17.193:57773
W20260429 20:01:09.274760 5418 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:09.275046 5418 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:09.275098 5418 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:09.278616 5418 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:09.278697 5418 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:09.278774 5418 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:01:09.282856 5418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:09.284168 5418 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:09.285409 5418 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:09.293043 5424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.293205 5423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.293191 5426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:09.294405 5418 server_base.cc:1061] running on GCE node
I20260429 20:01:09.295167 5418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:09.295949 5418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:09.297159 5418 hybrid_clock.cc:648] HybridClock initialized: now 1777492869297109 us; error 70 us; skew 500 ppm
I20260429 20:01:09.299584 5418 webserver.cc:492] Webserver started at http://127.27.17.194:37149/ using document root <none> and password file <none>
I20260429 20:01:09.300211 5418 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:09.300303 5418 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:09.300567 5418 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:09.302382 5418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "c43c9ab6980c4ad7b2092512b15e1d81"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "5a06087fa776e404a6effd5e10358e07"
server_key_iv: "45c95a51cd787eda6a2c73e43ffcefaa"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.302974 5418 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "c43c9ab6980c4ad7b2092512b15e1d81"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "5a06087fa776e404a6effd5e10358e07"
server_key_iv: "45c95a51cd787eda6a2c73e43ffcefaa"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.306807 5418 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260429 20:01:09.309566 5432 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.310796 5418 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:01:09.311008 5418 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "c43c9ab6980c4ad7b2092512b15e1d81"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "5a06087fa776e404a6effd5e10358e07"
server_key_iv: "45c95a51cd787eda6a2c73e43ffcefaa"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.311159 5418 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:09.334888 5418 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:09.335691 5418 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:09.335902 5418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:09.336515 5418 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:09.337543 5418 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:09.337617 5418 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.337689 5418 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:09.337736 5418 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.349153 5418 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:39071
I20260429 20:01:09.349174 5545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:39071 every 8 connection(s)
I20260429 20:01:09.350170 5418 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:01:09.355821 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5418
I20260429 20:01:09.355933 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:01:09.356278 27719 external_mini_cluster.cc:1468] Setting key 702c22558d5cce2e8cc5d7743a1fa42d
I20260429 20:01:09.358304 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45601
--builtin_ntp_servers=127.27.17.212:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260429 20:01:09.358392 5266 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.352090 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:40777 (local address 127.27.17.254:45601)
0429 20:01:09.352237 (+ 147us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:09.352240 (+ 3us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:09.352987 (+ 747us) server_negotiation.cc:408] Connection header received
0429 20:01:09.353870 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:09.353874 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:09.353942 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:09.354047 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:09.354935 (+ 888us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.355443 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:09.356220 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.356409 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:09.357837 (+ 1428us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:09.357863 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:09.357865 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:09.357877 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:09.357899 (+ 22us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:09.357907 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:09.357992 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:09.358181 (+ 189us) server_negotiation.cc:300] Negotiation successful
0429 20:01:09.358246 (+ 65us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260429 20:01:09.359103 5548 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.352274 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45601 (local address 127.27.17.194:40777)
0429 20:01:09.352845 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:09.352876 (+ 31us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:09.353676 (+ 800us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:09.354201 (+ 525us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:09.354210 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:09.354267 (+ 57us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:09.354761 (+ 494us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:09.354772 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.355599 (+ 827us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:09.355603 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:09.356105 (+ 502us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:09.356113 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.356371 (+ 258us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:09.357567 (+ 1196us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:01:09.357570 (+ 3us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:09.357607 (+ 37us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:01:09.357675 (+ 68us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:09.358009 (+ 334us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:09.358017 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:09.358022 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 20:01:09.358179 (+ 157us) client_negotiation.cc:241] Negotiation successful
0429 20:01:09.358361 (+ 182us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
I20260429 20:01:09.360690 5546 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45601
I20260429 20:01:09.360954 5546 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:09.361616 5546 heartbeater.cc:507] Master 127.27.17.254:45601 requested a full tablet report, sending...
I20260429 20:01:09.362852 5227 ts_manager.cc:194] Registered new tserver with Master: c43c9ab6980c4ad7b2092512b15e1d81 (127.27.17.194:39071)
I20260429 20:01:09.363492 5227 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.17.194:40777
W20260429 20:01:09.473240 5549 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:09.473495 5549 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:09.473546 5549 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:09.477313 5549 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:09.477396 5549 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:09.477505 5549 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:01:09.481885 5549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:09.483130 5549 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:09.484278 5549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:09.491338 5555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.491487 5554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.491346 5557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:09.491616 5549 server_base.cc:1061] running on GCE node
I20260429 20:01:09.491981 5549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:09.492600 5549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:09.493777 5549 hybrid_clock.cc:648] HybridClock initialized: now 1777492869493759 us; error 34 us; skew 500 ppm
I20260429 20:01:09.495944 5549 webserver.cc:492] Webserver started at http://127.27.17.195:33583/ using document root <none> and password file <none>
I20260429 20:01:09.496573 5549 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:09.496636 5549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:09.496891 5549 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:09.498641 5549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "214ec53794f44710b5e7dbd6768ee7fd"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "caa5ca8cfe0fb88bb7cfe195df82c353"
server_key_iv: "e6c02bffb7ee8dc98ac52a80e9b0386b"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.499202 5549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "214ec53794f44710b5e7dbd6768ee7fd"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "caa5ca8cfe0fb88bb7cfe195df82c353"
server_key_iv: "e6c02bffb7ee8dc98ac52a80e9b0386b"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.503070 5549 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260429 20:01:09.505699 5564 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.506989 5549 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:01:09.507105 5549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "214ec53794f44710b5e7dbd6768ee7fd"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "caa5ca8cfe0fb88bb7cfe195df82c353"
server_key_iv: "e6c02bffb7ee8dc98ac52a80e9b0386b"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.507261 5549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:09.520507 5549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:09.521210 5549 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:09.521420 5549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:09.522006 5549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:09.523072 5549 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:09.523124 5549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.523188 5549 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:09.523226 5549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.533062 5549 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:40575
I20260429 20:01:09.533083 5677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:40575 every 8 connection(s)
I20260429 20:01:09.534116 5549 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:01:09.535187 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5549
I20260429 20:01:09.535319 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:01:09.535586 27719 external_mini_cluster.cc:1468] Setting key e08fe0a6d42592a19de5cbbff5a8e979
I20260429 20:01:09.542559 5266 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.536095 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:59819 (local address 127.27.17.254:45601)
0429 20:01:09.536284 (+ 189us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:09.536289 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:09.537427 (+ 1138us) server_negotiation.cc:408] Connection header received
0429 20:01:09.538267 (+ 840us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:09.538270 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:09.538350 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:09.538422 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:09.539347 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.539850 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:09.540573 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.540725 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:09.542041 (+ 1316us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:09.542059 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:09.542061 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:09.542074 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:09.542094 (+ 20us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:09.542101 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:09.542173 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:09.542383 (+ 210us) server_negotiation.cc:300] Negotiation successful
0429 20:01:09.542434 (+ 51us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260429 20:01:09.543334 5680 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.536473 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45601 (local address 127.27.17.195:59819)
0429 20:01:09.537250 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:09.537291 (+ 41us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:09.538103 (+ 812us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:09.538572 (+ 469us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:09.538579 (+ 7us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:09.538636 (+ 57us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:09.539185 (+ 549us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:09.539197 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.540002 (+ 805us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:09.540017 (+ 15us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:09.540438 (+ 421us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:09.540445 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:09.540652 (+ 207us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:09.541804 (+ 1152us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:01:09.541806 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:09.541833 (+ 27us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:01:09.541883 (+ 50us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:09.542220 (+ 337us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:09.542228 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:09.542233 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 20:01:09.542374 (+ 141us) client_negotiation.cc:241] Negotiation successful
0429 20:01:09.542520 (+ 146us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":126,"threads_started":1}
I20260429 20:01:09.544579 5678 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45601
I20260429 20:01:09.544858 5678 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:09.545439 5678 heartbeater.cc:507] Master 127.27.17.254:45601 requested a full tablet report, sending...
I20260429 20:01:09.546695 5227 ts_manager.cc:194] Registered new tserver with Master: 214ec53794f44710b5e7dbd6768ee7fd (127.27.17.195:40575)
I20260429 20:01:09.547377 5227 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.27.17.195:59819
I20260429 20:01:09.549621 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 20:01:09.552706 5266 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:09.551742 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.27.17.254:45601)
0429 20:01:09.551918 (+ 176us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:09.551923 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:09.552041 (+ 118us) server_negotiation.cc:408] Connection header received
0429 20:01:09.552204 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:09.552207 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:09.552261 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:09.552349 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:09.552604 (+ 255us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52864: BlockingRecv error: recv got EOF from 127.0.0.1:52864 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260429 20:01:09.553237 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5287
I20260429 20:01:09.559859 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5418
I20260429 20:01:09.566135 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5549
I20260429 20:01:09.572410 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5196
2026-04-29T20:01:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (917 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-04-29T20:01:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:01:09Z Disabled control of system clock
I20260429 20:01:09.603082 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45193
--webserver_interface=127.27.17.254
--webserver_port=0
--builtin_ntp_servers=127.27.17.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:45193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 20:01:09.713660 5689 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:09.713943 5689 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:09.713994 5689 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:09.717847 5689 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:01:09.717942 5689 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:09.718077 5689 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:09.718122 5689 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:01:09.718142 5689 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:01:09.724114 5689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:45193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:09.725322 5689 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:09.726456 5689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:09.733254 5697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.733417 5694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.733201 5695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:09.733747 5689 server_base.cc:1061] running on GCE node
I20260429 20:01:09.734548 5689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:09.735745 5689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:09.737012 5689 hybrid_clock.cc:648] HybridClock initialized: now 1777492869736913 us; error 74 us; skew 500 ppm
I20260429 20:01:09.739359 5689 webserver.cc:492] Webserver started at http://127.27.17.254:33411/ using document root <none> and password file <none>
I20260429 20:01:09.739969 5689 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:09.740033 5689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:09.740208 5689 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:09.741995 5689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "356c598e3ac34961ae94bd018231412d"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "4489d7a3262fc2cba181c02a87dac104"
server_key_iv: "dd1cfb230522e30acf925d6f97f15f27"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.742488 5689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "356c598e3ac34961ae94bd018231412d"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "4489d7a3262fc2cba181c02a87dac104"
server_key_iv: "dd1cfb230522e30acf925d6f97f15f27"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.746104 5689 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260429 20:01:09.748955 5703 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.750252 5689 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:01:09.750453 5689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "356c598e3ac34961ae94bd018231412d"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "4489d7a3262fc2cba181c02a87dac104"
server_key_iv: "dd1cfb230522e30acf925d6f97f15f27"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.750586 5689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:09.756266 5689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:09.756994 5689 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:09.757213 5689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:09.767015 5689 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:45193
I20260429 20:01:09.766994 5755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:45193 every 8 connection(s)
I20260429 20:01:09.768177 5689 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:01:09.768878 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5689
I20260429 20:01:09.769034 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:01:09.769338 27719 external_mini_cluster.cc:1468] Setting key 6ea3fd890c05e8e18babea00adf0eb2e
I20260429 20:01:09.771804 5756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:01:09.775627 5759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:09.771040 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.27.17.254:45193)
0429 20:01:09.771688 (+ 648us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:09.771696 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:09.771734 (+ 38us) server_negotiation.cc:408] Connection header received
0429 20:01:09.772437 (+ 703us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:09.772455 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:09.772791 (+ 336us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:09.773089 (+ 298us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:09.774312 (+ 1223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:09.774318 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:09.774330 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:09.774343 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:09.774383 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:09.774395 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:09.774560 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:09.774800 (+ 240us) server_negotiation.cc:300] Negotiation successful
0429 20:01:09.774994 (+ 194us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":173,"threads_started":1}
I20260429 20:01:09.778335 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d: Bootstrap starting.
I20260429 20:01:09.780573 5756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d: Neither blocks nor log segments found. Creating new log.
I20260429 20:01:09.781553 5756 log.cc:826] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d: Log is configured to *not* fsync() on all Append() calls
I20260429 20:01:09.784117 5756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d: No bootstrap required, opened a new log
I20260429 20:01:09.786761 5756 raft_consensus.cc:359] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356c598e3ac34961ae94bd018231412d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45193 } }
I20260429 20:01:09.787052 5756 raft_consensus.cc:385] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:01:09.787106 5756 raft_consensus.cc:740] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356c598e3ac34961ae94bd018231412d, State: Initialized, Role: FOLLOWER
I20260429 20:01:09.787545 5756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356c598e3ac34961ae94bd018231412d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45193 } }
I20260429 20:01:09.787673 5756 raft_consensus.cc:399] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:01:09.787721 5756 raft_consensus.cc:493] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:01:09.787787 5756 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:01:09.789153 5756 raft_consensus.cc:515] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356c598e3ac34961ae94bd018231412d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45193 } }
I20260429 20:01:09.789592 5756 leader_election.cc:304] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356c598e3ac34961ae94bd018231412d; no voters:
I20260429 20:01:09.789928 5756 leader_election.cc:290] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:01:09.790117 5761 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:01:09.790426 5761 raft_consensus.cc:697] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [term 1 LEADER]: Becoming Leader. State: Replica: 356c598e3ac34961ae94bd018231412d, State: Running, Role: LEADER
I20260429 20:01:09.790755 5761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356c598e3ac34961ae94bd018231412d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45193 } }
I20260429 20:01:09.791240 5756 sys_catalog.cc:565] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:01:09.792538 5762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "356c598e3ac34961ae94bd018231412d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356c598e3ac34961ae94bd018231412d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45193 } } }
I20260429 20:01:09.792695 5762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [sys.catalog]: This master's current role is: LEADER
I20260429 20:01:09.793134 5763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 356c598e3ac34961ae94bd018231412d. Latest consensus state: current_term: 1 leader_uuid: "356c598e3ac34961ae94bd018231412d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356c598e3ac34961ae94bd018231412d" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 45193 } } }
I20260429 20:01:09.793267 5763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d [sys.catalog]: This master's current role is: LEADER
W20260429 20:01:09.797051 5776 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260429 20:01:09.797150 5776 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260429 20:01:09.797264 5770 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:01:09.798076 5770 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:01:09.804214 5770 catalog_manager.cc:1357] Generated new cluster ID: 9d67de975de34a78bc831f27bafd83d1
I20260429 20:01:09.804312 5770 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:01:09.813736 5770 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:01:09.814118 5770 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:01:09.826124 5770 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 356c598e3ac34961ae94bd018231412d: Generated new TSK 0
I20260429 20:01:09.826853 5770 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260429 20:01:09.831607 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45193
--builtin_ntp_servers=127.27.17.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260429 20:01:09.948050 5780 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:09.948330 5780 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:09.948388 5780 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:09.952222 5780 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:09.952488 5780 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:09.952582 5780 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:01:09.958001 5780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:09.959323 5780 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:09.960507 5780 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:09.968158 5785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.968158 5788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:09.968334 5786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:09.968714 5780 server_base.cc:1061] running on GCE node
I20260429 20:01:09.969168 5780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:09.969868 5780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:09.971011 5780 hybrid_clock.cc:648] HybridClock initialized: now 1777492869970982 us; error 45 us; skew 500 ppm
I20260429 20:01:09.973562 5780 webserver.cc:492] Webserver started at http://127.27.17.193:45803/ using document root <none> and password file <none>
I20260429 20:01:09.974208 5780 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:09.974273 5780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:09.974511 5780 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:09.976430 5780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "a0134de45b7b42d9aba8bff597bd1568"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "1a6584fe7fb8c6879ef67185896e0ed5"
server_key_iv: "ecb5b69cedaea30192227319a08922e5"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.976964 5780 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "a0134de45b7b42d9aba8bff597bd1568"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "1a6584fe7fb8c6879ef67185896e0ed5"
server_key_iv: "ecb5b69cedaea30192227319a08922e5"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.980647 5780 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 20:01:09.983048 5794 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.984037 5780 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:01:09.984179 5780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "a0134de45b7b42d9aba8bff597bd1568"
format_stamp: "Formatted at 2026-04-29 20:01:09 on dist-test-slave-01nr"
server_key: "1a6584fe7fb8c6879ef67185896e0ed5"
server_key_iv: "ecb5b69cedaea30192227319a08922e5"
server_key_version: "encryptionkey@0"
I20260429 20:01:09.984293 5780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:09.989181 5780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:09.989763 5780 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:09.989946 5780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:09.990537 5780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:09.991562 5780 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:09.991643 5780 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:09.991731 5780 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:09.991775 5780 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.001297 5780 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:35481
I20260429 20:01:10.001391 5907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:35481 every 8 connection(s)
I20260429 20:01:10.002424 5780 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:01:10.008697 5759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:10.004366 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:34017 (local address 127.27.17.254:45193)
0429 20:01:10.004550 (+ 184us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:10.004554 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:10.005492 (+ 938us) server_negotiation.cc:408] Connection header received
0429 20:01:10.006387 (+ 895us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:10.006391 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:10.006442 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:10.006526 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:10.007854 (+ 1328us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:10.007858 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:10.007861 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:10.007873 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:10.007897 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:10.007906 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:10.008003 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:10.008467 (+ 464us) server_negotiation.cc:300] Negotiation successful
0429 20:01:10.008511 (+ 44us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260429 20:01:10.008785 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5780
I20260429 20:01:10.009017 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:01:10.009526 27719 external_mini_cluster.cc:1468] Setting key 304faed45592ecadb4dc5bafa34424ff
I20260429 20:01:10.009820 5910 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:10.004653 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45193 (local address 127.27.17.193:34017)
0429 20:01:10.005296 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:10.005335 (+ 39us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:10.006156 (+ 821us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:10.006727 (+ 571us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:10.006736 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:10.006807 (+ 71us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:10.007550 (+ 743us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:01:10.007552 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:10.007581 (+ 29us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:01:10.007638 (+ 57us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:10.008204 (+ 566us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:10.008218 (+ 14us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:10.008223 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 20:01:10.008484 (+ 261us) client_negotiation.cc:241] Negotiation successful
0429 20:01:10.008629 (+ 145us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260429 20:01:10.011687 5908 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45193
I20260429 20:01:10.011984 5908 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:10.012156 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45193
--builtin_ntp_servers=127.27.17.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260429 20:01:10.012521 5908 heartbeater.cc:507] Master 127.27.17.254:45193 requested a full tablet report, sending...
I20260429 20:01:10.014648 5718 ts_manager.cc:194] Registered new tserver with Master: a0134de45b7b42d9aba8bff597bd1568 (127.27.17.193:35481)
W20260429 20:01:10.119799 5911 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:10.120059 5911 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:10.120114 5911 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:10.123809 5911 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:10.123893 5911 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:10.124017 5911 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:01:10.128639 5911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:10.130045 5911 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:10.131348 5911 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:10.138868 5917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:10.139178 5919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:10.139211 5916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:10.139389 5911 server_base.cc:1061] running on GCE node
I20260429 20:01:10.140048 5911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:10.140647 5911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:10.141832 5911 hybrid_clock.cc:648] HybridClock initialized: now 1777492870141804 us; error 45 us; skew 500 ppm
I20260429 20:01:10.144007 5911 webserver.cc:492] Webserver started at http://127.27.17.194:33191/ using document root <none> and password file <none>
I20260429 20:01:10.144630 5911 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:10.144691 5911 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:10.144905 5911 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:10.146780 5911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "476034fd75dd4bc6ba068d3bc4b0d7a4"
format_stamp: "Formatted at 2026-04-29 20:01:10 on dist-test-slave-01nr"
server_key: "835ef3930318c1c8589b4c35e9bb025b"
server_key_iv: "2d732bf9591e666461a09a20d787b564"
server_key_version: "encryptionkey@0"
I20260429 20:01:10.147365 5911 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "476034fd75dd4bc6ba068d3bc4b0d7a4"
format_stamp: "Formatted at 2026-04-29 20:01:10 on dist-test-slave-01nr"
server_key: "835ef3930318c1c8589b4c35e9bb025b"
server_key_iv: "2d732bf9591e666461a09a20d787b564"
server_key_version: "encryptionkey@0"
I20260429 20:01:10.151085 5911 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 20:01:10.153752 5925 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.155115 5911 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:01:10.155272 5911 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "476034fd75dd4bc6ba068d3bc4b0d7a4"
format_stamp: "Formatted at 2026-04-29 20:01:10 on dist-test-slave-01nr"
server_key: "835ef3930318c1c8589b4c35e9bb025b"
server_key_iv: "2d732bf9591e666461a09a20d787b564"
server_key_version: "encryptionkey@0"
I20260429 20:01:10.155397 5911 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:10.160578 5911 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:10.161204 5911 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:10.161401 5911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:10.161969 5911 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:10.163077 5911 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:10.163146 5911 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.163218 5911 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:10.163260 5911 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.175124 5911 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:35675
I20260429 20:01:10.175143 6038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:35675 every 8 connection(s)
I20260429 20:01:10.176136 5911 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:01:10.179526 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 5911
I20260429 20:01:10.179657 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:01:10.179963 27719 external_mini_cluster.cc:1468] Setting key a974d9b92932ebe272b1661fc3912871
I20260429 20:01:10.181936 5759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:10.177976 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:55115 (local address 127.27.17.254:45193)
0429 20:01:10.178137 (+ 161us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:10.178141 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:10.179018 (+ 877us) server_negotiation.cc:408] Connection header received
0429 20:01:10.179932 (+ 914us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:10.179936 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:10.179989 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:10.180136 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:10.181389 (+ 1253us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:10.181392 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:10.181394 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:10.181404 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:10.181425 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:10.181432 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:10.181528 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:10.181787 (+ 259us) server_negotiation.cc:300] Negotiation successful
0429 20:01:10.181826 (+ 39us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 20:01:10.182381 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45193
--builtin_ntp_servers=127.27.17.212:39131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260429 20:01:10.182775 6041 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:10.178245 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45193 (local address 127.27.17.194:55115)
0429 20:01:10.178831 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:10.178863 (+ 32us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:10.179735 (+ 872us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:10.180396 (+ 661us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:10.180405 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:10.180468 (+ 63us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:10.181134 (+ 666us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:01:10.181136 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:10.181160 (+ 24us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:01:10.181209 (+ 49us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:10.181526 (+ 317us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:10.181535 (+ 9us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:10.181544 (+ 9us) client_negotiation.cc:770] Sending connection context
0429 20:01:10.181794 (+ 250us) client_negotiation.cc:241] Negotiation successful
0429 20:01:10.181908 (+ 114us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":97,"threads_started":1}
I20260429 20:01:10.184278 6039 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45193
I20260429 20:01:10.184578 6039 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:10.185045 6039 heartbeater.cc:507] Master 127.27.17.254:45193 requested a full tablet report, sending...
I20260429 20:01:10.186301 5718 ts_manager.cc:194] Registered new tserver with Master: 476034fd75dd4bc6ba068d3bc4b0d7a4 (127.27.17.194:35675)
W20260429 20:01:10.291201 6042 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:10.291461 6042 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:10.291560 6042 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:10.295612 6042 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:10.295728 6042 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:10.295847 6042 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:01:10.300606 6042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:39131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:45193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:10.301916 6042 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:10.303181 6042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:10.310840 6047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:10.310945 6050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:10.310840 6048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:10.311147 6042 server_base.cc:1061] running on GCE node
I20260429 20:01:10.311717 6042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:10.312359 6042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:10.313561 6042 hybrid_clock.cc:648] HybridClock initialized: now 1777492870313547 us; error 27 us; skew 500 ppm
I20260429 20:01:10.315557 6042 webserver.cc:492] Webserver started at http://127.27.17.195:44305/ using document root <none> and password file <none>
I20260429 20:01:10.316148 6042 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:10.316236 6042 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:10.316463 6042 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:10.318337 6042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "6ca46092f9254196b26b3b871480c205"
format_stamp: "Formatted at 2026-04-29 20:01:10 on dist-test-slave-01nr"
server_key: "cfb09f3b79aacffb2049d4abbddc9799"
server_key_iv: "1041d17fd574ec93136b25ad5c06e892"
server_key_version: "encryptionkey@0"
I20260429 20:01:10.318884 6042 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "6ca46092f9254196b26b3b871480c205"
format_stamp: "Formatted at 2026-04-29 20:01:10 on dist-test-slave-01nr"
server_key: "cfb09f3b79aacffb2049d4abbddc9799"
server_key_iv: "1041d17fd574ec93136b25ad5c06e892"
server_key_version: "encryptionkey@0"
I20260429 20:01:10.322682 6042 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260429 20:01:10.325323 6056 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.326517 6042 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:01:10.326648 6042 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "6ca46092f9254196b26b3b871480c205"
format_stamp: "Formatted at 2026-04-29 20:01:10 on dist-test-slave-01nr"
server_key: "cfb09f3b79aacffb2049d4abbddc9799"
server_key_iv: "1041d17fd574ec93136b25ad5c06e892"
server_key_version: "encryptionkey@0"
I20260429 20:01:10.326761 6042 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:10.331931 6042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:10.332545 6042 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:10.332727 6042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:10.333300 6042 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:10.334287 6042 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:10.334336 6042 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.334400 6042 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:10.334432 6042 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:10.345321 6042 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:38247
I20260429 20:01:10.345480 6170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:38247 every 8 connection(s)
I20260429 20:01:10.346959 6042 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:01:10.349109 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 6042
I20260429 20:01:10.349213 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:01:10.349448 27719 external_mini_cluster.cc:1468] Setting key e59ab5115380e5d10a63fe8197f6bdb3
I20260429 20:01:10.354596 5759 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:10.350387 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:47139 (local address 127.27.17.254:45193)
0429 20:01:10.350566 (+ 179us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:10.350571 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:10.351731 (+ 1160us) server_negotiation.cc:408] Connection header received
0429 20:01:10.352601 (+ 870us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:10.352605 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:10.352649 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:10.352730 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:10.354021 (+ 1291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:10.354025 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:10.354027 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0429 20:01:10.354038 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:10.354061 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0429 20:01:10.354076 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:10.354172 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:10.354427 (+ 255us) server_negotiation.cc:300] Negotiation successful
0429 20:01:10.354468 (+ 41us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260429 20:01:10.355367 6173 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:10.350798 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:45193 (local address 127.27.17.195:47139)
0429 20:01:10.351577 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:10.351623 (+ 46us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:10.352355 (+ 732us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:10.352888 (+ 533us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:10.352897 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:10.352965 (+ 68us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:10.353716 (+ 751us) client_negotiation.cc:624] Initiating SASL PLAIN handshake
0429 20:01:10.353718 (+ 2us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:10.353746 (+ 28us) client_negotiation.cc:815] callback for SASL_CB_AUTHNAME
0429 20:01:10.353804 (+ 58us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:10.354191 (+ 387us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:10.354199 (+ 8us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:10.354204 (+ 5us) client_negotiation.cc:770] Sending connection context
0429 20:01:10.354413 (+ 209us) client_negotiation.cc:241] Negotiation successful
0429 20:01:10.354519 (+ 106us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":130,"threads_started":1}
I20260429 20:01:10.356557 6171 heartbeater.cc:344] Connected to a master server at 127.27.17.254:45193
I20260429 20:01:10.356851 6171 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:10.357297 6171 heartbeater.cc:507] Master 127.27.17.254:45193 requested a full tablet report, sending...
I20260429 20:01:10.358410 5718 ts_manager.cc:194] Registered new tserver with Master: 6ca46092f9254196b26b3b871480c205 (127.27.17.195:38247)
I20260429 20:01:10.364094 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20260429 20:01:10.367805 5759 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:01:10.366192 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:35890 (local address 127.27.17.254:45193)
0429 20:01:10.366445 (+ 253us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:10.366451 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:10.366610 (+ 159us) server_negotiation.cc:408] Connection header received
0429 20:01:10.367000 (+ 390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:10.367006 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:10.367062 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:10.367166 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:10.367643 (+ 477us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35890: BlockingRecv error: recv got EOF from 127.0.0.1:35890 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
I20260429 20:01:10.368068 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5780
I20260429 20:01:10.376536 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5911
I20260429 20:01:10.382434 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 6042
I20260429 20:01:10.388131 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 5689
2026-04-29T20:01:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (815 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:01:10 dist-test-slave-01nr krb5kdc[6181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:01:10 dist-test-slave-01nr krb5kdc[6181](info): set up 2 sockets
Apr 29 20:01:10 dist-test-slave-01nr krb5kdc[6181](info): commencing operation
krb5kdc: starting...
W20260429 20:01:12.480652 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.070s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:01:12 dist-test-slave-01nr krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:01:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:01:12.643786 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:46179
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:46179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:01:12.757867 6198 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:12.758176 6198 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:12.758268 6198 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:12.761883 6198 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:01:12.762012 6198 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:12.762054 6198 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:12.762099 6198 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:01:12.762140 6198 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:01:12.767060 6198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:46179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:46179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:12.768527 6198 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:12.769482 6198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:12.776613 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:12.776619 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:12.776788 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:12.777072 6198 server_base.cc:1061] running on GCE node
I20260429 20:01:12.777920 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:12.779116 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:12.780357 6198 hybrid_clock.cc:648] HybridClock initialized: now 1777492872780297 us; error 54 us; skew 500 ppm
Apr 29 20:01:12 dist-test-slave-01nr krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492872, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:01:12.783613 6198 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:01:12.784833 6198 webserver.cc:492] Webserver started at http://127.27.17.254:46353/ using document root <none> and password file <none>
I20260429 20:01:12.785449 6198 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:12.785526 6198 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:12.785749 6198 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:12.787638 6198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "a449727cd41d4702bc36c79902a3157e"
format_stamp: "Formatted at 2026-04-29 20:01:12 on dist-test-slave-01nr"
server_key: "7c3adecc396cb94d14ed17062277ea4b"
server_key_iv: "fd296fa7e48fce1a81f641039b6ff233"
server_key_version: "encryptionkey@0"
I20260429 20:01:12.788193 6198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "a449727cd41d4702bc36c79902a3157e"
format_stamp: "Formatted at 2026-04-29 20:01:12 on dist-test-slave-01nr"
server_key: "7c3adecc396cb94d14ed17062277ea4b"
server_key_iv: "fd296fa7e48fce1a81f641039b6ff233"
server_key_version: "encryptionkey@0"
I20260429 20:01:12.792300 6198 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260429 20:01:12.795069 6213 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:12.796183 6198 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:01:12.796409 6198 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "a449727cd41d4702bc36c79902a3157e"
format_stamp: "Formatted at 2026-04-29 20:01:12 on dist-test-slave-01nr"
server_key: "7c3adecc396cb94d14ed17062277ea4b"
server_key_iv: "fd296fa7e48fce1a81f641039b6ff233"
server_key_version: "encryptionkey@0"
I20260429 20:01:12.796548 6198 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:12.806752 6198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:12.810163 6198 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:12.810384 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:12.818282 6267 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:46179 every 8 connection(s)
I20260429 20:01:12.818279 6198 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:46179
I20260429 20:01:12.819383 6198 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:01:12.820596 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 6198
I20260429 20:01:12.820737 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:01:12.821007 27719 external_mini_cluster.cc:1468] Setting key 5610f4e6134693673ec73d2c085dc061
I20260429 20:01:12.822583 6268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:01:12.828524 6268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e: Bootstrap starting.
Apr 29 20:01:12 dist-test-slave-01nr krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:01:12.831190 6268 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e: Neither blocks nor log segments found. Creating new log.
I20260429 20:01:12.832074 6268 log.cc:826] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e: Log is configured to *not* fsync() on all Append() calls
I20260429 20:01:12.834327 6268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e: No bootstrap required, opened a new log
I20260429 20:01:12.834604 6271 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:12.822319 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.27.17.254:46179)
0429 20:01:12.822802 (+ 483us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:12.822812 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:12.822845 (+ 33us) server_negotiation.cc:408] Connection header received
0429 20:01:12.823577 (+ 732us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:12.823596 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:12.823879 (+ 283us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:12.824212 (+ 333us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:12.825181 (+ 969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:12.825956 (+ 775us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:12.826656 (+ 700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:12.826984 (+ 328us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:12.829435 (+ 2451us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:12.829497 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:12.829510 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:01:12.829565 (+ 55us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:12.831502 (+ 1937us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:12.832022 (+ 520us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:12.832028 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:12.832033 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:12.832125 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:12.832604 (+ 479us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:12.832608 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:12.832609 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:12.833056 (+ 447us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:12.833225 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:12.833636 (+ 411us) server_negotiation.cc:300] Negotiation successful
0429 20:01:12.833937 (+ 301us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":132,"threads_started":1}
I20260429 20:01:12.837098 6268 raft_consensus.cc:359] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a449727cd41d4702bc36c79902a3157e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46179 } }
I20260429 20:01:12.837309 6268 raft_consensus.cc:385] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:01:12.837347 6268 raft_consensus.cc:740] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a449727cd41d4702bc36c79902a3157e, State: Initialized, Role: FOLLOWER
I20260429 20:01:12.837806 6268 consensus_queue.cc:260] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a449727cd41d4702bc36c79902a3157e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46179 } }
I20260429 20:01:12.837934 6268 raft_consensus.cc:399] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:01:12.838002 6268 raft_consensus.cc:493] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:01:12.838093 6268 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:01:12.839123 6268 raft_consensus.cc:515] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a449727cd41d4702bc36c79902a3157e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46179 } }
I20260429 20:01:12.839485 6268 leader_election.cc:304] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a449727cd41d4702bc36c79902a3157e; no voters:
I20260429 20:01:12.839816 6268 leader_election.cc:290] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:01:12.840025 6273 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:01:12.840279 6273 raft_consensus.cc:697] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [term 1 LEADER]: Becoming Leader. State: Replica: a449727cd41d4702bc36c79902a3157e, State: Running, Role: LEADER
I20260429 20:01:12.840605 6273 consensus_queue.cc:237] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a449727cd41d4702bc36c79902a3157e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46179 } }
I20260429 20:01:12.841065 6268 sys_catalog.cc:565] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:01:12.842506 6275 sys_catalog.cc:455] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a449727cd41d4702bc36c79902a3157e. Latest consensus state: current_term: 1 leader_uuid: "a449727cd41d4702bc36c79902a3157e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a449727cd41d4702bc36c79902a3157e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46179 } } }
I20260429 20:01:12.842691 6275 sys_catalog.cc:458] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [sys.catalog]: This master's current role is: LEADER
I20260429 20:01:12.842674 6274 sys_catalog.cc:455] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a449727cd41d4702bc36c79902a3157e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a449727cd41d4702bc36c79902a3157e" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46179 } } }
I20260429 20:01:12.843127 6274 sys_catalog.cc:458] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e [sys.catalog]: This master's current role is: LEADER
I20260429 20:01:12.843847 6282 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:01:12.846593 6282 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:01:12.852034 6282 catalog_manager.cc:1357] Generated new cluster ID: d0df2251bbac48b68b61549d662a5433
I20260429 20:01:12.852134 6282 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:01:12.860175 6282 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:01:12.861168 6282 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:01:12.869045 6282 catalog_manager.cc:6044] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e: Generated new TSK 0
I20260429 20:01:12.869714 6282 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:01:12.939330 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:46179
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:01:13.048611 6297 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:13.048856 6297 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:13.048913 6297 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:13.052383 6297 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:13.052461 6297 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:13.052589 6297 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:01:13.057274 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:13.058513 6297 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:13.059463 6297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:13.066751 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:13.066731 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:13.066807 6297 server_base.cc:1061] running on GCE node
W20260429 20:01:13.066710 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:13.067437 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:13.068043 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:13.069252 6297 hybrid_clock.cc:648] HybridClock initialized: now 1777492873069223 us; error 41 us; skew 500 ppm
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492873, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:01:13.072292 6297 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:01:13.073400 6297 webserver.cc:492] Webserver started at http://127.27.17.193:46241/ using document root <none> and password file <none>
I20260429 20:01:13.073944 6297 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:13.073993 6297 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:13.074157 6297 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:13.075917 6297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "43d92db394750baa4a23efa01f049d78"
server_key_iv: "673881f973edec3033896aa0b962a183"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.076367 6297 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "43d92db394750baa4a23efa01f049d78"
server_key_iv: "673881f973edec3033896aa0b962a183"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.079802 6297 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 20:01:13.082062 6312 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.083374 6297 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260429 20:01:13.083549 6297 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "43d92db394750baa4a23efa01f049d78"
server_key_iv: "673881f973edec3033896aa0b962a183"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.083663 6297 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:13.104950 6297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:13.108212 6297 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:13.108443 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:13.109059 6297 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:13.110184 6297 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:13.110235 6297 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.110297 6297 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:13.110332 6297 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.120180 6297 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:40893
I20260429 20:01:13.120215 6429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:40893 every 8 connection(s)
I20260429 20:01:13.121110 6297 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:01:13.126117 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 6297
I20260429 20:01:13.126253 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:01:13.126525 27719 external_mini_cluster.cc:1468] Setting key 69f30799be5f21806009c58a352eb752
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492873, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:01:13.134160 6271 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.122920 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:35733 (local address 127.27.17.254:46179)
0429 20:01:13.123077 (+ 157us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.123081 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.123737 (+ 656us) server_negotiation.cc:408] Connection header received
0429 20:01:13.124624 (+ 887us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.124628 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.124699 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.124795 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:13.125978 (+ 1183us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.126625 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.127365 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.127519 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.130466 (+ 2947us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:13.130490 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:13.130493 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:01:13.130521 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:13.132153 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.132702 (+ 549us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.132706 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.132708 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.132764 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.133076 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.133079 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.133081 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.133252 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:13.133360 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.133859 (+ 499us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.133994 (+ 135us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260429 20:01:13.134817 6432 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.123160 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46179 (local address 127.27.17.193:35733)
0429 20:01:13.123599 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.123633 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.124395 (+ 762us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.124960 (+ 565us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.124969 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.125344 (+ 375us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:13.125823 (+ 479us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.125835 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.126759 (+ 924us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.126763 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.127213 (+ 450us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.127243 (+ 30us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.127460 (+ 217us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.128629 (+ 1169us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:13.128648 (+ 19us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:13.130288 (+ 1640us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:13.132293 (+ 2005us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:01:13.132301 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:01:13.132314 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:01:13.132587 (+ 273us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:01:13.132863 (+ 276us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:01:13.132867 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:01:13.132871 (+ 4us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:01:13.132970 (+ 99us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:01:13.133368 (+ 398us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:13.133378 (+ 10us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:13.133610 (+ 232us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.133824 (+ 214us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.134067 (+ 243us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
I20260429 20:01:13.136032 6430 heartbeater.cc:344] Connected to a master server at 127.27.17.254:46179
I20260429 20:01:13.136380 6430 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:13.136917 6430 heartbeater.cc:507] Master 127.27.17.254:46179 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
I20260429 20:01:13.138530 6231 ts_manager.cc:194] Registered new tserver with Master: a2e2ecd7fc7e41178d602ac5cf14b4d4 (127.27.17.193:40893)
I20260429 20:01:13.139957 6231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:35733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:01:13.180017 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:46179
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:01:13.287396 6438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:13.287701 6438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:13.287812 6438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:13.291294 6438 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:13.291373 6438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:13.291501 6438 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:01:13.296181 6438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:13.297380 6438 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:13.298264 6438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:13.305192 6443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:13.305200 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:13.305253 6438 server_base.cc:1061] running on GCE node
W20260429 20:01:13.305276 6446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:13.305910 6438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:13.306519 6438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:13.307714 6438 hybrid_clock.cc:648] HybridClock initialized: now 1777492873307690 us; error 41 us; skew 500 ppm
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492873, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:01:13.310792 6438 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:01:13.312068 6438 webserver.cc:492] Webserver started at http://127.27.17.194:46009/ using document root <none> and password file <none>
I20260429 20:01:13.312686 6438 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:13.312738 6438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:13.312968 6438 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:13.314735 6438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "7bcf8714461964f5cb6ad6393776f8cb"
server_key_iv: "03a25739ecfdc48f7f68d8da817f6761"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.315301 6438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "7bcf8714461964f5cb6ad6393776f8cb"
server_key_iv: "03a25739ecfdc48f7f68d8da817f6761"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.318980 6438 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260429 20:01:13.321267 6453 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.322337 6438 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260429 20:01:13.322510 6438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "7bcf8714461964f5cb6ad6393776f8cb"
server_key_iv: "03a25739ecfdc48f7f68d8da817f6761"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.322633 6438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:13.340389 6438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:13.343562 6438 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:13.343776 6438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:13.344403 6438 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:13.345319 6438 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:13.345402 6438 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.345474 6438 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:13.345527 6438 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.354957 6438 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:43169
I20260429 20:01:13.354987 6567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:43169 every 8 connection(s)
I20260429 20:01:13.355962 6438 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:01:13.357082 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 6438
I20260429 20:01:13.357206 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:01:13.357482 27719 external_mini_cluster.cc:1468] Setting key 51e5ad3e6c334edfe140fc131d5cd2e1
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492873, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.17.195@KRBTEST.COM; defaulting to no policy
I20260429 20:01:13.372876 6271 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.358139 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:33627 (local address 127.27.17.254:46179)
0429 20:01:13.358291 (+ 152us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.358296 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.359031 (+ 735us) server_negotiation.cc:408] Connection header received
0429 20:01:13.360133 (+ 1102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.360138 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.360198 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.360306 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:13.362066 (+ 1760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.362889 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.363781 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.363968 (+ 187us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.367632 (+ 3664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:13.367660 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:13.367664 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:01:13.367699 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:13.369857 (+ 2158us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.370752 (+ 895us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.370756 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.370758 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.370842 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.371421 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.371428 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.371432 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.371704 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:13.371804 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.372544 (+ 740us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.372704 (+ 160us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.195@KRBTEST.COM" created.
I20260429 20:01:13.373950 6570 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.358442 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46179 (local address 127.27.17.194:33627)
0429 20:01:13.358872 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.358944 (+ 72us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.359743 (+ 799us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.360464 (+ 721us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.360473 (+ 9us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.361115 (+ 642us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:13.361823 (+ 708us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.361848 (+ 25us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.363073 (+ 1225us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.363078 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.363644 (+ 566us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.363654 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.364181 (+ 527us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.365434 (+ 1253us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:13.365458 (+ 24us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:13.367416 (+ 1958us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:13.370044 (+ 2628us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:01:13.370051 (+ 7us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:01:13.370064 (+ 13us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:01:13.370525 (+ 461us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:01:13.371053 (+ 528us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:01:13.371057 (+ 4us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:01:13.371058 (+ 1us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:01:13.371257 (+ 199us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:01:13.371892 (+ 635us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:13.371898 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:13.372226 (+ 328us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.372490 (+ 264us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.372799 (+ 309us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":127,"threads_started":1}
I20260429 20:01:13.375288 6568 heartbeater.cc:344] Connected to a master server at 127.27.17.254:46179
I20260429 20:01:13.375527 6568 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:13.376013 6568 heartbeater.cc:507] Master 127.27.17.254:46179 requested a full tablet report, sending...
I20260429 20:01:13.377358 6231 ts_manager.cc:194] Registered new tserver with Master: 7839e2383cdf44a6b1ee6b3d9bc69a73 (127.27.17.194:43169)
I20260429 20:01:13.378041 6231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.194@KRBTEST.COM'} at 127.27.17.194:33627
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:01:13.424113 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.195:0
--local_ip_for_outbound_sockets=127.27.17.195
--webserver_interface=127.27.17.195
--webserver_port=0
--tserver_master_addrs=127.27.17.254:46179
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:40157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:01:13.529685 6575 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:01:13.529968 6575 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:01:13.530059 6575 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:01:13.533492 6575 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:01:13.533641 6575 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:01:13.533756 6575 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.195
I20260429 20:01:13.538249 6575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:40157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.17.195
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:46179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.195
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:01:13.539393 6575 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:01:13.540267 6575 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:01:13.546998 6580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:13.547142 6575 server_base.cc:1061] running on GCE node
W20260429 20:01:13.546998 6581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:01:13.546998 6583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:01:13.547729 6575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:01:13.548316 6575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:01:13.549558 6575 hybrid_clock.cc:648] HybridClock initialized: now 1777492873549533 us; error 40 us; skew 500 ppm
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492873, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:01:13.553232 6575 init.cc:377] Logged in from keytab as kudu/127.27.17.195@KRBTEST.COM (short username kudu)
I20260429 20:01:13.554316 6575 webserver.cc:492] Webserver started at http://127.27.17.195:39121/ using document root <none> and password file <none>
I20260429 20:01:13.554868 6575 fs_manager.cc:362] Metadata directory not provided
I20260429 20:01:13.554981 6575 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:01:13.555168 6575 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260429 20:01:13.556891 6575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/instance:
uuid: "5b10eb64235044efb3f6d3ad3e587b04"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "541fdcc5c6d7d6788ed70ab2d28e9f3b"
server_key_iv: "636795c628aa168998faac13ca56f5a4"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.557384 6575 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance:
uuid: "5b10eb64235044efb3f6d3ad3e587b04"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "541fdcc5c6d7d6788ed70ab2d28e9f3b"
server_key_iv: "636795c628aa168998faac13ca56f5a4"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.561048 6575 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260429 20:01:13.563493 6590 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.564770 6575 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260429 20:01:13.564944 6575 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
uuid: "5b10eb64235044efb3f6d3ad3e587b04"
format_stamp: "Formatted at 2026-04-29 20:01:13 on dist-test-slave-01nr"
server_key: "541fdcc5c6d7d6788ed70ab2d28e9f3b"
server_key_iv: "636795c628aa168998faac13ca56f5a4"
server_key_version: "encryptionkey@0"
I20260429 20:01:13.565070 6575 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:01:13.581526 6575 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:01:13.584612 6575 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:01:13.584836 6575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:01:13.585427 6575 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:01:13.586421 6575 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:01:13.586498 6575 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.586602 6575 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:01:13.586652 6575 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:01:13.596367 6575 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.195:39023
I20260429 20:01:13.596444 6703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.195:39023 every 8 connection(s)
I20260429 20:01:13.597379 6575 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/data/info.pb
I20260429 20:01:13.600652 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 6575
I20260429 20:01:13.600809 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/instance
I20260429 20:01:13.601131 27719 external_mini_cluster.cc:1468] Setting key 7e35f6efecfdfc52a4fd2098f8a4b511
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492873, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.195@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:01:13.612186 6271 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.599166 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.195:41347 (local address 127.27.17.254:46179)
0429 20:01:13.599347 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.599352 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.600153 (+ 801us) server_negotiation.cc:408] Connection header received
0429 20:01:13.601407 (+ 1254us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.601410 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.601573 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.601688 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:13.603285 (+ 1597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.603891 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.604812 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.604962 (+ 150us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.608116 (+ 3154us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:13.608139 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:13.608142 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:01:13.608174 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:13.609806 (+ 1632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.610494 (+ 688us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.610497 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.610498 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.610550 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.611001 (+ 451us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.611004 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.611005 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.611176 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:13.611269 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.611902 (+ 633us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.612045 (+ 143us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260429 20:01:13.612738 6706 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.599551 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46179 (local address 127.27.17.195:41347)
0429 20:01:13.599990 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.600025 (+ 35us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.601090 (+ 1065us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.601879 (+ 789us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.601892 (+ 13us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.602411 (+ 519us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:01:13.603134 (+ 723us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.603151 (+ 17us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.604035 (+ 884us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.604042 (+ 7us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.604675 (+ 633us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.604684 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.604858 (+ 174us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.605971 (+ 1113us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:01:13.605992 (+ 21us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:01:13.607902 (+ 1910us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:01:13.610058 (+ 2156us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:01:13.610066 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:01:13.610088 (+ 22us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:01:13.610357 (+ 269us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:01:13.610655 (+ 298us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:01:13.610661 (+ 6us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:01:13.610667 (+ 6us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:01:13.610821 (+ 154us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:01:13.611342 (+ 521us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:01:13.611348 (+ 6us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:01:13.611590 (+ 242us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.611778 (+ 188us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.611993 (+ 215us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":106,"threads_started":1}
I20260429 20:01:13.614028 6704 heartbeater.cc:344] Connected to a master server at 127.27.17.254:46179
I20260429 20:01:13.614316 6704 heartbeater.cc:461] Registering TS with master...
I20260429 20:01:13.615031 6704 heartbeater.cc:507] Master 127.27.17.254:46179 requested a full tablet report, sending...
I20260429 20:01:13.616180 6231 ts_manager.cc:194] Registered new tserver with Master: 5b10eb64235044efb3f6d3ad3e587b04 (127.27.17.195:39023)
I20260429 20:01:13.616832 6231 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.195@KRBTEST.COM'} at 127.27.17.195:41347
I20260429 20:01:13.628062 27719 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260429 20:01:13.637440 6271 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.630333 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.27.17.254:46179)
0429 20:01:13.630497 (+ 164us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.630504 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.630522 (+ 18us) server_negotiation.cc:408] Connection header received
0429 20:01:13.630569 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.630578 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.630623 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.630715 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:01:13.631712 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.632220 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.632899 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.633129 (+ 230us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.634247 (+ 1118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:01:13.634274 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:01:13.634276 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:01:13.634305 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:01:13.635852 (+ 1547us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.636269 (+ 417us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.636273 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.636275 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.636337 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:01:13.636590 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:01:13.636593 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:01:13.636595 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:01:13.636781 (+ 186us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:01:13.636919 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.637148 (+ 229us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.637258 (+ 110us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260429 20:01:13.640403 6231 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260429 20:01:13.643002 6231 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260429 20:01:13.656109 6715 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.651612 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:40893 (local address 127.0.0.1:48804)
0429 20:01:13.651962 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.652057 (+ 95us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.652198 (+ 141us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.652891 (+ 693us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.652896 (+ 5us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.652918 (+ 22us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:01:13.653273 (+ 355us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.653282 (+ 9us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.654636 (+ 1354us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.654640 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.655637 (+ 997us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.655644 (+ 7us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.655743 (+ 99us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.655799 (+ 56us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.655889 (+ 90us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.655958 (+ 69us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260429 20:01:13.656435 6718 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.652321 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:43169 (local address 127.0.0.1:60656)
0429 20:01:13.653187 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.653199 (+ 12us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.653278 (+ 79us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.653692 (+ 414us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.653695 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.653714 (+ 19us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:01:13.653962 (+ 248us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.653968 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.655200 (+ 1232us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.655204 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.656115 (+ 911us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.656125 (+ 10us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.656211 (+ 86us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.656226 (+ 15us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.656265 (+ 39us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.656312 (+ 47us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":84,"threads_started":1}
I20260429 20:01:13.656766 6716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.651780 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.27.17.193:40893)
0429 20:01:13.652379 (+ 599us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.652389 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.652415 (+ 26us) server_negotiation.cc:408] Connection header received
0429 20:01:13.652511 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.652519 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.652733 (+ 214us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.652868 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:01:13.653414 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.654515 (+ 1101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.655793 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.656334 (+ 541us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.656452 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.656524 (+ 72us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.656620 (+ 96us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":103,"threads_started":1}
I20260429 20:01:13.656839 6717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.652194 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:39023 (local address 127.0.0.1:60714)
0429 20:01:13.653094 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.653128 (+ 34us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.653217 (+ 89us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.653890 (+ 673us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.653893 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.653908 (+ 15us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:01:13.654123 (+ 215us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.654128 (+ 5us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.655412 (+ 1284us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.655416 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.656462 (+ 1046us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.656470 (+ 8us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.656613 (+ 143us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.656626 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.656672 (+ 46us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.656724 (+ 52us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":107,"threads_started":1}
I20260429 20:01:13.657580 6720 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.652731 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.27.17.194:43169)
0429 20:01:13.653126 (+ 395us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.653132 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.653277 (+ 145us) server_negotiation.cc:408] Connection header received
0429 20:01:13.653394 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.653400 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.653548 (+ 148us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.653700 (+ 152us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:01:13.654087 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.655068 (+ 981us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.656682 (+ 1614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.657221 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.657310 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.657368 (+ 58us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.657436 (+ 68us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":134,"threads_started":1}
I20260429 20:01:13.657886 6719 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.652533 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60714 (local address 127.27.17.195:39023)
0429 20:01:13.653521 (+ 988us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.653527 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.653546 (+ 19us) server_negotiation.cc:408] Connection header received
0429 20:01:13.653627 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.653632 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.653760 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.653890 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:01:13.654233 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.655261 (+ 1028us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.657154 (+ 1893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.657531 (+ 377us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.657598 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.657680 (+ 82us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.657763 (+ 83us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":107,"threads_started":1}
I20260429 20:01:13.659888 6502 tablet_service.cc:1511] Processing CreateTablet for tablet 38492f755bd4457293aab52f63566f8b (DEFAULT_TABLE table=test-table [id=9b9bdb565292472fb68ed899b13b1bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:01:13.659922 6359 tablet_service.cc:1511] Processing CreateTablet for tablet 38492f755bd4457293aab52f63566f8b (DEFAULT_TABLE table=test-table [id=9b9bdb565292472fb68ed899b13b1bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:01:13.660239 6638 tablet_service.cc:1511] Processing CreateTablet for tablet 38492f755bd4457293aab52f63566f8b (DEFAULT_TABLE table=test-table [id=9b9bdb565292472fb68ed899b13b1bae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260429 20:01:13.660874 6502 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38492f755bd4457293aab52f63566f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:01:13.660887 6359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38492f755bd4457293aab52f63566f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:01:13.661033 6638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38492f755bd4457293aab52f63566f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:01:13.667102 6721 tablet_bootstrap.cc:492] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: Bootstrap starting.
I20260429 20:01:13.667582 6722 tablet_bootstrap.cc:492] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4: Bootstrap starting.
I20260429 20:01:13.667840 6723 tablet_bootstrap.cc:492] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Bootstrap starting.
I20260429 20:01:13.670163 6722 tablet_bootstrap.cc:654] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4: Neither blocks nor log segments found. Creating new log.
I20260429 20:01:13.670163 6723 tablet_bootstrap.cc:654] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Neither blocks nor log segments found. Creating new log.
I20260429 20:01:13.671195 6722 log.cc:826] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4: Log is configured to *not* fsync() on all Append() calls
I20260429 20:01:13.671195 6723 log.cc:826] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Log is configured to *not* fsync() on all Append() calls
I20260429 20:01:13.672264 6721 tablet_bootstrap.cc:654] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: Neither blocks nor log segments found. Creating new log.
I20260429 20:01:13.673000 6721 log.cc:826] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: Log is configured to *not* fsync() on all Append() calls
I20260429 20:01:13.673375 6722 tablet_bootstrap.cc:492] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4: No bootstrap required, opened a new log
I20260429 20:01:13.673374 6723 tablet_bootstrap.cc:492] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: No bootstrap required, opened a new log
I20260429 20:01:13.673592 6722 ts_tablet_manager.cc:1403] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 20:01:13.673592 6723 ts_tablet_manager.cc:1403] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260429 20:01:13.674796 6721 tablet_bootstrap.cc:492] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: No bootstrap required, opened a new log
I20260429 20:01:13.675050 6721 ts_tablet_manager.cc:1403] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260429 20:01:13.677078 6722 raft_consensus.cc:359] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.677358 6722 raft_consensus.cc:385] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:01:13.677428 6722 raft_consensus.cc:740] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e2ecd7fc7e41178d602ac5cf14b4d4, State: Initialized, Role: FOLLOWER
I20260429 20:01:13.677335 6723 raft_consensus.cc:359] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.677538 6723 raft_consensus.cc:385] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:01:13.677593 6723 raft_consensus.cc:740] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b10eb64235044efb3f6d3ad3e587b04, State: Initialized, Role: FOLLOWER
I20260429 20:01:13.677960 6723 consensus_queue.cc:260] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.677961 6722 consensus_queue.cc:260] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.678700 6721 raft_consensus.cc:359] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.678947 6721 raft_consensus.cc:385] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:01:13.678987 6723 ts_tablet_manager.cc:1434] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260429 20:01:13.679009 6721 raft_consensus.cc:740] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7839e2383cdf44a6b1ee6b3d9bc69a73, State: Initialized, Role: FOLLOWER
I20260429 20:01:13.679028 6430 heartbeater.cc:499] Master 127.27.17.254:46179 was elected leader, sending a full tablet report...
I20260429 20:01:13.679266 6704 heartbeater.cc:499] Master 127.27.17.254:46179 was elected leader, sending a full tablet report...
I20260429 20:01:13.679487 6721 consensus_queue.cc:260] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.680119 6568 heartbeater.cc:499] Master 127.27.17.254:46179 was elected leader, sending a full tablet report...
I20260429 20:01:13.681545 6722 ts_tablet_manager.cc:1434] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260429 20:01:13.680408 6721 ts_tablet_manager.cc:1434] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20260429 20:01:13.711210 6729 raft_consensus.cc:493] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260429 20:01:13.711458 6729 raft_consensus.cc:515] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.712757 6729 leader_election.cc:290] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2e2ecd7fc7e41178d602ac5cf14b4d4 (127.27.17.193:40893), 5b10eb64235044efb3f6d3ad3e587b04 (127.27.17.195:39023)
I20260429 20:01:13.716888 6570 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.713120 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.195:39023 (local address 127.27.17.194:49191)
0429 20:01:13.713376 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.713394 (+ 18us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.713557 (+ 163us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.713926 (+ 369us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.713929 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.713972 (+ 43us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:01:13.714265 (+ 293us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.714271 (+ 6us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.715452 (+ 1181us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.715457 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.716438 (+ 981us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.716449 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.716585 (+ 136us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.716604 (+ 19us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.716659 (+ 55us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.716716 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"spinlock_wait_cycles":29696}
I20260429 20:01:13.717291 6730 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.713127 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:40893 (local address 127.27.17.194:54161)
0429 20:01:13.713570 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.713585 (+ 15us) client_negotiation.cc:175] Beginning negotiation
0429 20:01:13.713675 (+ 90us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:01:13.713926 (+ 251us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:01:13.713929 (+ 3us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:01:13.713972 (+ 43us) client_negotiation.cc:190] Negotiated authn=CERTIFICATE
0429 20:01:13.714265 (+ 293us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.714316 (+ 51us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.715452 (+ 1136us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.715457 (+ 5us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:01:13.716438 (+ 981us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:01:13.716449 (+ 11us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.716591 (+ 142us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.716604 (+ 13us) client_negotiation.cc:770] Sending connection context
0429 20:01:13.716659 (+ 55us) client_negotiation.cc:241] Negotiation successful
0429 20:01:13.716716 (+ 57us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":58,"spinlock_wait_cycles":4096,"thread_start_us":136,"threads_started":1}
I20260429 20:01:13.717361 6716 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.713218 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:54161 (local address 127.27.17.193:40893)
0429 20:01:13.713376 (+ 158us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.713382 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.713690 (+ 308us) server_negotiation.cc:408] Connection header received
0429 20:01:13.713761 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.713764 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.713812 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.713899 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:01:13.714439 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.715304 (+ 865us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.716582 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.717087 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.717128 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.717180 (+ 52us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.717240 (+ 60us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260429 20:01:13.717392 6719 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.713218 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.194:49191 (local address 127.27.17.195:39023)
0429 20:01:13.713399 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.713404 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.713461 (+ 57us) server_negotiation.cc:408] Connection header received
0429 20:01:13.713740 (+ 279us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.713744 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.713803 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.713899 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0429 20:01:13.714439 (+ 540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.715280 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.716674 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.717153 (+ 479us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.717182 (+ 29us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.717227 (+ 45us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.717275 (+ 48us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260429 20:01:13.718011 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38492f755bd4457293aab52f63566f8b" candidate_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" is_pre_election: true
I20260429 20:01:13.718012 6658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38492f755bd4457293aab52f63566f8b" candidate_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b10eb64235044efb3f6d3ad3e587b04" is_pre_election: true
I20260429 20:01:13.718297 6381 raft_consensus.cc:2468] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7839e2383cdf44a6b1ee6b3d9bc69a73 in term 0.
I20260429 20:01:13.718297 6658 raft_consensus.cc:2468] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7839e2383cdf44a6b1ee6b3d9bc69a73 in term 0.
I20260429 20:01:13.718802 6456 leader_election.cc:304] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b10eb64235044efb3f6d3ad3e587b04, 7839e2383cdf44a6b1ee6b3d9bc69a73; no voters:
I20260429 20:01:13.719110 6729 raft_consensus.cc:2804] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260429 20:01:13.719187 6729 raft_consensus.cc:493] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260429 20:01:13.719230 6729 raft_consensus.cc:3060] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:01:13.720194 6729 raft_consensus.cc:515] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.720577 6729 leader_election.cc:290] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [CANDIDATE]: Term 1 election: Requested vote from peers a2e2ecd7fc7e41178d602ac5cf14b4d4 (127.27.17.193:40893), 5b10eb64235044efb3f6d3ad3e587b04 (127.27.17.195:39023)
I20260429 20:01:13.720963 6658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38492f755bd4457293aab52f63566f8b" candidate_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b10eb64235044efb3f6d3ad3e587b04"
I20260429 20:01:13.721063 6381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38492f755bd4457293aab52f63566f8b" candidate_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4"
I20260429 20:01:13.721078 6658 raft_consensus.cc:3060] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:01:13.721222 6381 raft_consensus.cc:3060] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:01:13.722182 6381 raft_consensus.cc:2468] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7839e2383cdf44a6b1ee6b3d9bc69a73 in term 1.
I20260429 20:01:13.722185 6658 raft_consensus.cc:2468] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7839e2383cdf44a6b1ee6b3d9bc69a73 in term 1.
I20260429 20:01:13.722532 6456 leader_election.cc:304] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b10eb64235044efb3f6d3ad3e587b04, 7839e2383cdf44a6b1ee6b3d9bc69a73; no voters:
I20260429 20:01:13.722743 6729 raft_consensus.cc:2804] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:01:13.723073 6729 raft_consensus.cc:697] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 1 LEADER]: Becoming Leader. State: Replica: 7839e2383cdf44a6b1ee6b3d9bc69a73, State: Running, Role: LEADER
I20260429 20:01:13.723425 6729 consensus_queue.cc:237] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } }
I20260429 20:01:13.727018 6226 catalog_manager.cc:5671] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7839e2383cdf44a6b1ee6b3d9bc69a73 (127.27.17.194). New cstate: current_term: 1 leader_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7839e2383cdf44a6b1ee6b3d9bc69a73" member_type: VOTER last_known_addr { host: "127.27.17.194" port: 43169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 } health_report { overall_health: UNKNOWN } } }
I20260429 20:01:13.742141 6720 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.738893 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.27.17.194:43169)
0429 20:01:13.739096 (+ 203us) server_negotiation.cc:207] Beginning negotiation
0429 20:01:13.739103 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0429 20:01:13.739120 (+ 17us) server_negotiation.cc:408] Connection header received
0429 20:01:13.739211 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:01:13.739214 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:01:13.739267 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:01:13.739377 (+ 110us) server_negotiation.cc:227] Negotiated authn=TOKEN
0429 20:01:13.739745 (+ 368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.740250 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:01:13.740962 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:01:13.741164 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:01:13.741251 (+ 87us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
0429 20:01:13.741662 (+ 411us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
0429 20:01:13.741792 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
0429 20:01:13.741873 (+ 81us) server_negotiation.cc:300] Negotiation successful
0429 20:01:13.741926 (+ 53us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260429 20:01:13.748194 6381 raft_consensus.cc:1275] T 38492f755bd4457293aab52f63566f8b P a2e2ecd7fc7e41178d602ac5cf14b4d4 [term 1 FOLLOWER]: Refusing update from remote peer 7839e2383cdf44a6b1ee6b3d9bc69a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:01:13.748195 6658 raft_consensus.cc:1275] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 1 FOLLOWER]: Refusing update from remote peer 7839e2383cdf44a6b1ee6b3d9bc69a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260429 20:01:13.748982 6729 consensus_queue.cc:1048] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e2ecd7fc7e41178d602ac5cf14b4d4" member_type: VOTER last_known_addr { host: "127.27.17.193" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:01:13.749229 6731 consensus_queue.cc:1048] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b10eb64235044efb3f6d3ad3e587b04" member_type: VOTER last_known_addr { host: "127.27.17.195" port: 39023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260429 20:01:13.756973 6734 mvcc.cc:204] Tried to move back new op lower bound from 7280610810863976448 to 7280610810774056960. Current Snapshot: MvccSnapshot[applied={T|T < 7280610810863976448}]
I20260429 20:01:13.758147 6736 mvcc.cc:204] Tried to move back new op lower bound from 7280610810863976448 to 7280610810774056960. Current Snapshot: MvccSnapshot[applied={T|T < 7280610810863976448}]
I20260429 20:01:13.766469 6226 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:01:13.766685 6226 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36236:
table { table_name: "test-table" } modify_external_catalogs: true
I20260429 20:01:13.769789 6226 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a449727cd41d4702bc36c79902a3157e: Sending DeleteTablet for 3 replicas of tablet 38492f755bd4457293aab52f63566f8b
I20260429 20:01:13.770784 6638 tablet_service.cc:1558] Processing DeleteTablet for tablet 38492f755bd4457293aab52f63566f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:01:13 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:60714
I20260429 20:01:13.770817 6359 tablet_service.cc:1558] Processing DeleteTablet for tablet 38492f755bd4457293aab52f63566f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:01:13 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:48804
I20260429 20:01:13.770951 6502 tablet_service.cc:1558] Processing DeleteTablet for tablet 38492f755bd4457293aab52f63566f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-04-29 20:01:13 UTC) from {username='kudu', principal='kudu/127.27.17.254@KRBTEST.COM'} at 127.0.0.1:60656
I20260429 20:01:13.771520 6746 tablet_replica.cc:333] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: stopping tablet replica
I20260429 20:01:13.771703 6745 tablet_replica.cc:333] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: stopping tablet replica
I20260429 20:01:13.771823 6746 raft_consensus.cc:2243] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 1 LEADER]: Raft consensus shutting down.
I20260429 20:01:13.771973 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 6297
I20260429 20:01:13.772238 6746 raft_consensus.cc:2272] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:01:13.773738 6745 raft_consensus.cc:2243] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260429 20:01:13.774107 6745 raft_consensus.cc:2272] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260429 20:01:13.775827 6745 ts_tablet_manager.cc:1916] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:01:13.778692 6745 ts_tablet_manager.cc:1929] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260429 20:01:13.778816 6745 log.cc:1199] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Deleting WAL directory at /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1777492770945061-27719-0/minicluster-data/ts-2/wal/wals/38492f755bd4457293aab52f63566f8b
I20260429 20:01:13.779214 6745 ts_tablet_manager.cc:1950] T 38492f755bd4457293aab52f63566f8b P 5b10eb64235044efb3f6d3ad3e587b04: Deleting consensus metadata
I20260429 20:01:13.780308 6746 ts_tablet_manager.cc:1916] T 38492f755bd4457293aab52f63566f8b P 7839e2383cdf44a6b1ee6b3d9bc69a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260429 20:01:13.780709 6216 catalog_manager.cc:5002] TS 5b10eb64235044efb3f6d3ad3e587b04 (127.27.17.195:39023): tablet 38492f755bd4457293aab52f63566f8b (table test-table [id=9b9bdb565292472fb68ed899b13b1bae]) successfully deleted
W20260429 20:01:13.781279 6217 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.17.193:40893 (error 108)
I20260429 20:01:13.781594 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 6438
I20260429 20:01:13.783095 6717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.782657 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.193:40893 (local address 127.0.0.1:48818)
0429 20:01:13.782883 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.782980 (+ 97us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.193:40893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260429 20:01:13.783269 6217 catalog_manager.cc:4729] TS a2e2ecd7fc7e41178d602ac5cf14b4d4 (127.27.17.193:40893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 38492f755bd4457293aab52f63566f8b: Network error: Client connection negotiation failed: client connection to 127.27.17.193:40893: connect: Connection refused (error 111)
I20260429 20:01:13.788405 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 6575
I20260429 20:01:13.789449 6717 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:01:13.788684 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.194:43169 (local address 127.0.0.1:60680)
0429 20:01:13.789280 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0429 20:01:13.789334 (+ 54us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.194:43169: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477}
I20260429 20:01:13.794656 27719 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task05EAH3/build/debug/bin/kudu with pid 6198
2026-04-29T20:01:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3407 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6750](info): set up 2 sockets
Apr 29 20:01:13 dist-test-slave-01nr krb5kdc[6750](info): commencing operation
krb5kdc: starting...
W20260429 20:01:15.844338 27719 mini_kdc.cc:121] Time spent starting KDC: real 2.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/test-user.keytab.
Apr 29 20:01:15 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-04-29T20:01:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-04-29T20:01:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/rangeradmin_127.27.17.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/rangeradmin_127.27.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/rangerlookup_127.27.17.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/rangerlookup_127.27.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/HTTP_127.27.17.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/HTTP_127.27.17.212@KRBTEST.COM.keytab.
I20260429 20:01:16.044193 27719 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task05EAH3/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/postgres -l logfile start
2026-04-29 20:01:19.253 UTC [6781] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2026-04-29 20:01:19.253 UTC [6781] LOG: listening on IPv4 address "127.27.17.212", port 53603
2026-04-29 20:01:19.259 UTC [6781] LOG: listening on Unix socket "/tmp/.s.PGSQL.53603"
2026-04-29 20:01:19.265 UTC [6786] LOG: database system was shut down at 2026-04-29 20:01:17 UTC
2026-04-29 20:01:19.271 UTC [6781] LOG: database system is ready to accept connections
I20260429 20:01:21.208170 27719 mini_postgres.cc:96] Postgres bound to 53603
2026-04-29 20:01:21.214 UTC [6792] FATAL: database "slave" does not exist
127.27.17.212:53603 - accepting connections
I20260429 20:01:21.215637 27719 mini_ranger.cc:161] Starting Ranger...
I20260429 20:01:21.235016 27719 mini_ranger.cc:84] Created miniranger Postgres user
I20260429 20:01:21.308936 27719 mini_ranger.cc:87] Created ranger Postgres database
I20260429 20:01:21.309055 27719 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-admin
2026-04-29 20:01:21,611 [I] DB FLAVOR :POSTGRES
2026-04-29 20:01:21,611 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 20:01:21,611 [I] Checking connection..
2026-04-29 20:01:21,611 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-04-29 20:01:21,986 [I] Checking connection passed.
2026-04-29 20:01:21,986 [I] --------- Verifying version history table ---------
2026-04-29 20:01:21,986 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 20:01:22,355 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 20:01:22,356 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 20:01:22,712 [I] Table x_db_version_h does not exist in database ranger
2026-04-29 20:01:22,712 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-04-29 20:01:22,712 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-04-29 20:01:23.065 UTC [6908] WARNING: there is no transaction in progress
2026-04-29 20:01:23,091 [I] create_dbversion_catalog.sql file imported successfully
2026-04-29 20:01:23,091 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-04-29 20:01:23,443 [I] Table x_db_version_h already exists in database 'ranger'
2026-04-29 20:01:23,443 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 20:01:23,444 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-04-29 20:01:23,782 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-04-29 20:01:24,114 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-01nr.c.gcp-upstream.internal','N') ;"
2026-04-29 20:01:24,490 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 20:01:24,874 [I] Table x_portal_user does not exist in database ranger
2026-04-29 20:01:24,875 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 20:01:25,249 [I] Table x_policy_ref_group does not exist in database ranger
2026-04-29 20:01:25,249 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 20:01:25,639 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 20:01:26,040 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-04-29 20:01:26,040 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-04-29 20:01:27.019 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.031 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.045 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.057 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.069 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.083 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.136 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.142 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.153 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.163 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.173 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.182 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.191 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.197 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.201 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27.204 UTC [7107] WARNING: there is no transaction in progress
2026-04-29 20:01:27,726 [I] ranger_core_db_postgres.sql file imported successfully
2026-04-29 20:01:27,727 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-04-29 20:01:28,029 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-04-29 20:01:28,029 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-04-29 20:01:28,361 [I] Table x_portal_user already exists in database 'ranger'
2026-04-29 20:01:28,361 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-04-29 20:01:28,695 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-04-29 20:01:28,696 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-04-29 20:01:29,026 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-04-29 20:01:29,362 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-01nr.c.gcp-upstream.internal';"
2026-04-29 20:01:29,674 [I] CORE_DB_SCHEMA import status has been updated
2026-04-29 20:01:29,675 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-04-29 20:01:30,056 [I] DB_PATCHES have already been applied
I20260429 20:01:30.061888 27719 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-admin:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*:/tmp/dist-test-task05EAH3/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260429 20:01:30.061995 27719 mini_ranger.cc:193] Using host: 127.27.17.212
I20260429 20:01:30.065189 27719 mini_ranger.cc:239] Ranger admin URL: http://127.27.17.212:44801
Apr 29, 2026 8:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 8:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 8:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-admin/ews/webapp
Apr 29, 2026 8:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Apr 29, 2026 8:01:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-admin/ews/webapp] .....
Apr 29, 2026 8:01:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-admin/ews/webapp].
Apr 29, 2026 8:01:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260429 20:01:32.376094 27719 mini_ranger.cc:160] Time spent starting Ranger: real 11.160s user 0.003s sys 0.008s
Apr 29, 2026 8:01:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Apr 29, 2026 8:01:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.UserREST
Apr 29, 2026 8:01:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Apr 29, 2026 8:01:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Apr 29, 2026 8:01:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260429 20:01:59.068617 27719 mini_ranger.cc:273] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.17.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.17.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/rangerkms_127.27.17.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.17.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/rangerkms_127.27.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.17.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.17.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/HTTP_127.27.17.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.17.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/HTTP_127.27.17.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Apr 29 20:01:59 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492919, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260429 20:01:59.176127 27719 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260429 20:01:59.188422 27719 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260429 20:01:59.318706 27719 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260429 20:01:59.318804 27719 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms
I20260429 20:01:59.318843 27719 mini_ranger_kms.cc:227] Using postgres at 127.27.17.212:53603
2026-04-29 20:01:59,546 [I] DB FLAVOR :POSTGRES
2026-04-29 20:01:59,547 [I] --------- Verifying Ranger DB connection ---------
2026-04-29 20:01:59,547 [I] Checking connection
2026-04-29 20:01:59,547 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-04-29 20:01:59,995 [I] connection success
2026-04-29 20:01:59,995 [I] --------- Verifying Ranger DB tables ---------
2026-04-29 20:01:59,996 [I] Verifying table ranger_masterkey in database rangerkms
2026-04-29 20:01:59,996 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-04-29 20:02:00,445 [I] Table ranger_masterkey does not exist in database rangerkms
2026-04-29 20:02:00,446 [I] --------- Importing Ranger Core DB Schema ---------
2026-04-29 20:02:00,446 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-04-29 20:02:00,446 [JISQL] /usr/lib/jvm/temurin-17-jdk-amd64/bin/java -cp /tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.17.212:53603/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-04-29 20:02:00,853 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260429 20:02:01.455646 27719 mini_ranger_kms.cc:326] Created kms service
I20260429 20:02:01.674607 27719 mini_ranger_kms.cc:342] Created kudu user
I20260429 20:02:01.730141 27719 mini_ranger_kms.cc:359] Created rangerkms user
Apr 29 20:02:01 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492919, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.17.212@KRBTEST.COM
I20260429 20:02:02.255434 27719 mini_ranger_kms.cc:400] Added ranger policy
I20260429 20:02:02.255626 27719 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms:/tmp/dist-test-task05EAH3/build/debug/bin/postgresql.jar:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task05EAH3/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/temurin-17-jdk-amd64/lib/*
I20260429 20:02:02.255650 27719 mini_ranger_kms.cc:240] Using host: 127.27.17.212
I20260429 20:02:02.260020 27719 mini_ranger_kms.cc:292] Ranger KMS PID: 7407
I20260429 20:02:02.260143 27719 mini_ranger_kms.cc:293] Ranger KMS URL: http://127.27.17.212:60891
Apr 29, 2026 8:02:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Apr 29, 2026 8:02:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Apr 29, 2026 8:02:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Apr 29, 2026 8:02:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/ews/webapp] .....
Apr 29, 2026 8:02:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-kms/ews/webapp].
Apr 29, 2026 8:02:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260429 20:02:04.591838 27719 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.416s user 0.011s sys 0.187s
I20260429 20:02:04.591976 27719 mini_ranger_kms.cc:413] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260429 20:02:04.592059 27719 mini_ranger_kms.cc:417] 127.27.17.212:60891/kms/v1/keys
Apr 29 20:02:05 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777492925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.17.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Apr 29 20:02:05 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777492925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.17.212@KRBTEST.COM for HTTP/127.27.17.212@KRBTEST.COM
Apr 29 20:02:06 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1777492925, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.17.212@KRBTEST.COM for HTTP/127.27.17.212@KRBTEST.COM
Apr 29 20:02:08 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:02:08.636319 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:46305
--webserver_interface=127.27.17.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.17.254:46305
--ranger_config_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.17.212:60891/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:02:08.743564 7475 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:02:08.743891 7475 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:02:08.743995 7475 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:02:08.747536 7475 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260429 20:02:08.747648 7475 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:02:08.747702 7475 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:02:08.747747 7475 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260429 20:02:08.747789 7475 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260429 20:02:08.752547 7475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.17.212:60891/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.17.254:46305
--ranger_config_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.17.254:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.17.254
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:02:08.753702 7475 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:02:08.754597 7475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:02:08.761183 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:02:08.761299 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:02:08.761476 7475 server_base.cc:1061] running on GCE node
W20260429 20:02:08.761304 7483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:02:08.762127 7475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:02:08.763159 7475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:02:08.764319 7475 hybrid_clock.cc:648] HybridClock initialized: now 1777492928764282 us; error 59 us; skew 500 ppm
Apr 29 20:02:08 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492928, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:02:08.767655 7475 init.cc:377] Logged in from keytab as kudu/127.27.17.254@KRBTEST.COM (short username kudu)
I20260429 20:02:08.768810 7475 webserver.cc:492] Webserver started at http://127.27.17.254:33605/ using document root <none> and password file <none>
I20260429 20:02:08.769400 7475 fs_manager.cc:362] Metadata directory not provided
I20260429 20:02:08.769479 7475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:02:08.769694 7475 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 20:02:08 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492928, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.254@KRBTEST.COM for HTTP/127.27.17.212@KRBTEST.COM
I20260429 20:02:08.814334 7475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data/instance:
uuid: "ac9f9c98cc06446995e76368f63bcc21"
format_stamp: "Formatted at 2026-04-29 20:02:08 on dist-test-slave-01nr"
server_key: "cea715c0d4c5a7093b5407d5d59e08c5"
server_key_iv: "bab6009a3edeeb3f808c4fdbd2916930"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:08.815037 7475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal/instance:
uuid: "ac9f9c98cc06446995e76368f63bcc21"
format_stamp: "Formatted at 2026-04-29 20:02:08 on dist-test-slave-01nr"
server_key: "cea715c0d4c5a7093b5407d5d59e08c5"
server_key_iv: "bab6009a3edeeb3f808c4fdbd2916930"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:08.818760 7475 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260429 20:02:08.836921 7491 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:02:08.838752 7475 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:02:08.838969 7475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal
uuid: "ac9f9c98cc06446995e76368f63bcc21"
format_stamp: "Formatted at 2026-04-29 20:02:08 on dist-test-slave-01nr"
server_key: "cea715c0d4c5a7093b5407d5d59e08c5"
server_key_iv: "bab6009a3edeeb3f808c4fdbd2916930"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:08.839119 7475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:02:08.856179 7475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:02:08.865718 7475 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:02:08.865952 7475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:02:08.874569 7475 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.254:46305
I20260429 20:02:08.874573 7543 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.254:46305 every 8 connection(s)
I20260429 20:02:08.875789 7475 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/data/info.pb
I20260429 20:02:08.878924 7544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260429 20:02:08.884611 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 7475
I20260429 20:02:08.884789 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/wal/instance
I20260429 20:02:08.884830 7544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21: Bootstrap starting.
I20260429 20:02:08.885079 27719 external_mini_cluster.cc:1468] Setting key e48d3feafeef8d23117e2dffffb422ef
I20260429 20:02:08.887559 7544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21: Neither blocks nor log segments found. Creating new log.
I20260429 20:02:08.888306 7544 log.cc:826] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21: Log is configured to *not* fsync() on all Append() calls
I20260429 20:02:08.890146 7544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21: No bootstrap required, opened a new log
Apr 29 20:02:08 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:02:08.894369 7544 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9f9c98cc06446995e76368f63bcc21" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46305 } }
I20260429 20:02:08.894657 7544 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260429 20:02:08.894734 7544 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9f9c98cc06446995e76368f63bcc21, State: Initialized, Role: FOLLOWER
I20260429 20:02:08.895352 7544 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9f9c98cc06446995e76368f63bcc21" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46305 } }
I20260429 20:02:08.895749 7544 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260429 20:02:08.895841 7544 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260429 20:02:08.895949 7544 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 0 FOLLOWER]: Advancing to term 1
I20260429 20:02:08.897169 7544 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9f9c98cc06446995e76368f63bcc21" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46305 } }
I20260429 20:02:08.897837 7544 leader_election.cc:304] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac9f9c98cc06446995e76368f63bcc21; no voters:
I20260429 20:02:08.898217 7544 leader_election.cc:290] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260429 20:02:08.898389 7549 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 1 FOLLOWER]: Leader election won for term 1
I20260429 20:02:08.898658 7549 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [term 1 LEADER]: Becoming Leader. State: Replica: ac9f9c98cc06446995e76368f63bcc21, State: Running, Role: LEADER
I20260429 20:02:08.899053 7549 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9f9c98cc06446995e76368f63bcc21" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46305 } }
I20260429 20:02:08.899595 7544 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [sys.catalog]: configured and running, proceeding with master startup.
I20260429 20:02:08.900530 7547 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:08.886536 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.0.0.1:50532 (local address 127.27.17.254:46305)
0429 20:02:08.887070 (+ 534us) server_negotiation.cc:207] Beginning negotiation
0429 20:02:08.887080 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0429 20:02:08.887116 (+ 36us) server_negotiation.cc:408] Connection header received
0429 20:02:08.887792 (+ 676us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:02:08.887810 (+ 18us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:02:08.888098 (+ 288us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:02:08.888449 (+ 351us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:02:08.889439 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:02:08.890232 (+ 793us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:02:08.891183 (+ 951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:02:08.891488 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:02:08.894104 (+ 2616us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:02:08.894132 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:02:08.894148 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:02:08.894189 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:02:08.896758 (+ 2569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:02:08.897274 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:02:08.897283 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:02:08.897290 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:02:08.897387 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:02:08.897870 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:02:08.897874 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:02:08.897876 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:02:08.898409 (+ 533us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:02:08.898630 (+ 221us) server_negotiation.cc:1036] Waiting for connection context
0429 20:02:08.899494 (+ 864us) server_negotiation.cc:300] Negotiation successful
0429 20:02:08.899720 (+ 226us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":138,"threads_started":1}
I20260429 20:02:08.900836 7551 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac9f9c98cc06446995e76368f63bcc21. Latest consensus state: current_term: 1 leader_uuid: "ac9f9c98cc06446995e76368f63bcc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9f9c98cc06446995e76368f63bcc21" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46305 } } }
I20260429 20:02:08.901105 7551 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [sys.catalog]: This master's current role is: LEADER
I20260429 20:02:08.901633 7558 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260429 20:02:08.901772 7550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac9f9c98cc06446995e76368f63bcc21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac9f9c98cc06446995e76368f63bcc21" member_type: VOTER last_known_addr { host: "127.27.17.254" port: 46305 } } }
I20260429 20:02:08.901854 7550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21 [sys.catalog]: This master's current role is: LEADER
I20260429 20:02:08.902052 7544 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260429 20:02:08.908375 7558 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260429 20:02:08.916239 7558 catalog_manager.cc:1357] Generated new cluster ID: 25ef18b6b8fe4f14a38ca3f0ab95a616
I20260429 20:02:08.916335 7558 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260429 20:02:08.952646 7558 catalog_manager.cc:1380] Generated new certificate authority record
I20260429 20:02:08.953969 7558 catalog_manager.cc:1514] Loading token signing keys...
I20260429 20:02:08.975239 7558 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ac9f9c98cc06446995e76368f63bcc21: Generated new TSK 0
I20260429 20:02:08.976223 7558 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:02:09.748555 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.17.212:60891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.193:0
--local_ip_for_outbound_sockets=127.27.17.193
--webserver_interface=127.27.17.193
--webserver_port=0
--tserver_master_addrs=127.27.17.254:46305
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260429 20:02:09.904335 7603 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:02:09.904799 7603 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:02:09.904933 7603 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:02:09.910449 7603 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:02:09.910667 7603 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:02:09.910847 7603 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.193
I20260429 20:02:09.918028 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.17.212:60891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.17.193
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.193
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:02:09.919957 7603 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:02:09.921274 7603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:02:09.931833 7611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:02:09.933444 7613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:02:09.934440 7603 server_base.cc:1061] running on GCE node
W20260429 20:02:09.934790 7610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:02:09.935415 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:02:09.936133 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:02:09.937332 7603 hybrid_clock.cc:648] HybridClock initialized: now 1777492929937311 us; error 40 us; skew 500 ppm
Apr 29 20:02:09 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492929, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:02:09.941460 7603 init.cc:377] Logged in from keytab as kudu/127.27.17.193@KRBTEST.COM (short username kudu)
I20260429 20:02:09.943293 7603 webserver.cc:492] Webserver started at http://127.27.17.193:40433/ using document root <none> and password file <none>
I20260429 20:02:09.944283 7603 fs_manager.cc:362] Metadata directory not provided
I20260429 20:02:09.944437 7603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:02:09.944949 7603 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 20:02:09 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492929, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for HTTP/127.27.17.212@KRBTEST.COM
I20260429 20:02:09.965072 7603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data/instance:
uuid: "5f363193ade845139fb54bb73d7d94da"
format_stamp: "Formatted at 2026-04-29 20:02:09 on dist-test-slave-01nr"
server_key: "1f1532c3e94fe8a86919078ca2fbc547"
server_key_iv: "0cb5c182d0b2112c2b84730077d9d1c1"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:09.965935 7603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance:
uuid: "5f363193ade845139fb54bb73d7d94da"
format_stamp: "Formatted at 2026-04-29 20:02:09 on dist-test-slave-01nr"
server_key: "1f1532c3e94fe8a86919078ca2fbc547"
server_key_iv: "0cb5c182d0b2112c2b84730077d9d1c1"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:09.983093 7603 fs_manager.cc:696] Time spent creating directory manager: real 0.017s user 0.004s sys 0.000s
I20260429 20:02:10.000977 7622 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:02:10.002493 7603 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260429 20:02:10.002674 7603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal
uuid: "5f363193ade845139fb54bb73d7d94da"
format_stamp: "Formatted at 2026-04-29 20:02:09 on dist-test-slave-01nr"
server_key: "1f1532c3e94fe8a86919078ca2fbc547"
server_key_iv: "0cb5c182d0b2112c2b84730077d9d1c1"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:10.002818 7603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:02:10.029042 7603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:02:10.033277 7603 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:02:10.033682 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:02:10.034540 7603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:02:10.035902 7603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:02:10.036031 7603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260429 20:02:10.036157 7603 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:02:10.036244 7603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:02:10.051282 7603 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.193:34203
I20260429 20:02:10.052903 7603 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/data/info.pb
I20260429 20:02:10.054937 7737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.193:34203 every 8 connection(s)
I20260429 20:02:10.063146 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 7603
I20260429 20:02:10.063305 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-0/wal/instance
I20260429 20:02:10.063622 27719 external_mini_cluster.cc:1468] Setting key 353f18e9c365c28243332da688d1ef6d
Apr 29 20:02:10 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492929, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.193@KRBTEST.COM for kudu/127.27.17.254@KRBTEST.COM
I20260429 20:02:10.074476 7740 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:10.057207 (+ 0us) reactor.cc:678] Submitting negotiation task for server connection from 127.27.17.193:40783 (local address 127.27.17.254:46305)
0429 20:02:10.057726 (+ 519us) server_negotiation.cc:207] Beginning negotiation
0429 20:02:10.057731 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0429 20:02:10.058286 (+ 555us) server_negotiation.cc:408] Connection header received
0429 20:02:10.059580 (+ 1294us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0429 20:02:10.059584 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0429 20:02:10.059658 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0429 20:02:10.059735 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
0429 20:02:10.061562 (+ 1827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:02:10.062420 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0429 20:02:10.063939 (+ 1519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0429 20:02:10.064182 (+ 243us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:02:10.069541 (+ 5359us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0429 20:02:10.069574 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0429 20:02:10.069578 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0429 20:02:10.069615 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
0429 20:02:10.071684 (+ 2069us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:02:10.072449 (+ 765us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:02:10.072454 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:02:10.072456 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:02:10.072517 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0429 20:02:10.072893 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0429 20:02:10.072897 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0429 20:02:10.072899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
0429 20:02:10.073105 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0429 20:02:10.073208 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
0429 20:02:10.074109 (+ 901us) server_negotiation.cc:300] Negotiation successful
0429 20:02:10.074261 (+ 152us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":111,"threads_started":1}
I20260429 20:02:10.076385 7741 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:10.057637 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.193:40783)
0429 20:02:10.058130 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:10.058173 (+ 43us) client_negotiation.cc:175] Beginning negotiation
0429 20:02:10.059263 (+ 1090us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:02:10.060215 (+ 952us) client_negotiation.cc:272] Received NEGOTIATE NegotiatePB response
0429 20:02:10.060227 (+ 12us) client_negotiation.cc:395] Received NEGOTIATE response from server
0429 20:02:10.060640 (+ 413us) client_negotiation.cc:190] Negotiated authn=SASL
0429 20:02:10.061337 (+ 697us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:02:10.061356 (+ 19us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:02:10.062606 (+ 1250us) client_negotiation.cc:272] Received TLS_HANDSHAKE NegotiatePB response
0429 20:02:10.062610 (+ 4us) client_negotiation.cc:528] Received TLS_HANDSHAKE response from server
0429 20:02:10.063580 (+ 970us) client_negotiation.cc:515] Sending TLS_HANDSHAKE message to server
0429 20:02:10.063592 (+ 12us) client_negotiation.cc:253] Sending TLS_HANDSHAKE NegotiatePB request
0429 20:02:10.066030 (+ 2438us) client_negotiation.cc:549] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0429 20:02:10.067179 (+ 1149us) client_negotiation.cc:624] Initiating SASL GSSAPI handshake
0429 20:02:10.067210 (+ 31us) client_negotiation.cc:657] calling sasl_client_start()
0429 20:02:10.069248 (+ 2038us) client_negotiation.cc:253] Sending SASL_INITIATE NegotiatePB request
0429 20:02:10.071894 (+ 2646us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:02:10.071902 (+ 8us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:02:10.071917 (+ 15us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:02:10.072294 (+ 377us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:02:10.072638 (+ 344us) client_negotiation.cc:272] Received SASL_CHALLENGE NegotiatePB response
0429 20:02:10.072641 (+ 3us) client_negotiation.cc:695] Received SASL_CHALLENGE response from server
0429 20:02:10.072643 (+ 2us) client_negotiation.cc:761] calling sasl_client_step()
0429 20:02:10.072779 (+ 136us) client_negotiation.cc:253] Sending SASL_RESPONSE NegotiatePB request
0429 20:02:10.073297 (+ 518us) client_negotiation.cc:272] Received SASL_SUCCESS NegotiatePB response
0429 20:02:10.073313 (+ 16us) client_negotiation.cc:716] Received SASL_SUCCESS response from server
0429 20:02:10.073772 (+ 459us) client_negotiation.cc:770] Sending connection context
0429 20:02:10.075054 (+ 1282us) client_negotiation.cc:241] Negotiation successful
0429 20:02:10.075346 (+ 292us) negotiation.cc:326] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":147,"threads_started":1}
I20260429 20:02:10.078337 7738 heartbeater.cc:344] Connected to a master server at 127.27.17.254:46305
I20260429 20:02:10.078653 7738 heartbeater.cc:461] Registering TS with master...
I20260429 20:02:10.079412 7738 heartbeater.cc:507] Master 127.27.17.254:46305 requested a full tablet report, sending...
I20260429 20:02:10.081497 7508 ts_manager.cc:194] Registered new tserver with Master: 5f363193ade845139fb54bb73d7d94da (127.27.17.193:34203)
I20260429 20:02:10.083238 7508 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.17.193@KRBTEST.COM'} at 127.27.17.193:40783
WARNING: no policy specified for kudu/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.17.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.27.17.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.17.194@KRBTEST.COM" created.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.27.17.254 from keytab /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:878)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:745)
at jdk.security.auth/com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:597)
at java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:679)
at java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:677)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:677)
at java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.27.17.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab.
I20260429 20:02:10.153017 27719 external_mini_cluster.cc:1366] Running /tmp/dist-test-task05EAH3/build/debug/bin/kudu
/tmp/dist-test-task05EAH3/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.17.212:60891/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.17.194:0
--local_ip_for_outbound_sockets=127.27.17.194
--webserver_interface=127.27.17.194
--webserver_port=0
--tserver_master_addrs=127.27.17.254:46305
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.27.17.212:46717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260429 20:02:10.169333 7595 server.cc:273] Received an EOF from the subprocess
F20260429 20:02:10.170840 7568 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7fb369101dcd google::LogMessage::Fail() at ??:0
@ 0x7fb369105b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb3691017cc google::LogMessage::Flush() at ??:0
@ 0x7fb369102f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb36af1a157 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7fb36af16f59 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7fb36af1ae85 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c977ae2244 std::function<>::operator()() at ??:0
@ 0x7fb36a6067ea kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb36af4a6db start_thread at ??:0
@ 0x7fb3682b861f clone at ??:0
W20260429 20:02:10.305471 7747 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260429 20:02:10.306145 7747 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260429 20:02:10.306283 7747 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260429 20:02:10.311685 7747 flags.cc:432] Enabled experimental flag: --rpc_trace_negotiation=true
W20260429 20:02:10.311790 7747 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260429 20:02:10.311910 7747 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.17.194
I20260429 20:02:10.318156 7747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.17.212:46717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.17.212:60891/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/krb5kdc/kudu.keytab
--principal=kudu/127.27.17.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.17.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.17.194
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.27.17.254:46305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.17.194
--log_dir=/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 8537ed5bb5e0e6ac1e481ff961e6b9d08cce7671
build type DEBUG
built by None at 29 Apr 2026 19:43:17 UTC on bdcb31816ec0
build id 11664
I20260429 20:02:10.319737 7747 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260429 20:02:10.320784 7747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260429 20:02:10.329762 7763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:02:10.329844 7766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260429 20:02:10.329758 7764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260429 20:02:10.330173 7747 server_base.cc:1061] running on GCE node
I20260429 20:02:10.330667 7747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260429 20:02:10.331445 7747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260429 20:02:10.332759 7747 hybrid_clock.cc:648] HybridClock initialized: now 1777492930332725 us; error 43 us; skew 500 ppm
Apr 29 20:02:10 dist-test-slave-01nr krb5kdc[6750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492930, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260429 20:02:10.337147 7747 init.cc:377] Logged in from keytab as kudu/127.27.17.194@KRBTEST.COM (short username kudu)
I20260429 20:02:10.338505 7747 webserver.cc:492] Webserver started at http://127.27.17.194:43081/ using document root <none> and password file <none>
I20260429 20:02:10.339226 7747 fs_manager.cc:362] Metadata directory not provided
I20260429 20:02:10.339308 7747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260429 20:02:10.339553 7747 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
Apr 29 20:02:10 dist-test-slave-01nr krb5kdc[6750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1777492930, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.17.194@KRBTEST.COM for HTTP/127.27.17.212@KRBTEST.COM
I20260429 20:02:10.358002 7747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data/instance:
uuid: "5c8e48663b7043f08955e6dec3320689"
format_stamp: "Formatted at 2026-04-29 20:02:10 on dist-test-slave-01nr"
server_key: "92e8031f43597c5dd0f95cbe06b2c422"
server_key_iv: "756369baba50c1854e91eb70e5852ca9"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:10.358858 7747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance:
uuid: "5c8e48663b7043f08955e6dec3320689"
format_stamp: "Formatted at 2026-04-29 20:02:10 on dist-test-slave-01nr"
server_key: "92e8031f43597c5dd0f95cbe06b2c422"
server_key_iv: "756369baba50c1854e91eb70e5852ca9"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:10.363171 7747 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260429 20:02:10.378680 7773 log_block_manager.cc:3869] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260429 20:02:10.379966 7747 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260429 20:02:10.380119 7747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data,/tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal
uuid: "5c8e48663b7043f08955e6dec3320689"
format_stamp: "Formatted at 2026-04-29 20:02:10 on dist-test-slave-01nr"
server_key: "92e8031f43597c5dd0f95cbe06b2c422"
server_key_iv: "756369baba50c1854e91eb70e5852ca9"
server_key_version: "kuduclusterkey@0"
I20260429 20:02:10.380261 7747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260429 20:02:10.390588 7747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260429 20:02:10.393635 7747 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260429 20:02:10.393854 7747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260429 20:02:10.394486 7747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260429 20:02:10.395557 7747 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260429 20:02:10.395637 7747 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260429 20:02:10.395706 7747 ts_tablet_manager.cc:616] Registered 0 tablets
I20260429 20:02:10.395759 7747 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260429 20:02:10.408222 7747 rpc_server.cc:307] RPC server started. Bound to: 127.27.17.194:40119
I20260429 20:02:10.408237 7886 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.27.17.194:40119 every 8 connection(s)
I20260429 20:02:10.409353 7747 server_base.cc:1193] Dumped server information to /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/data/info.pb
I20260429 20:02:10.411226 27719 external_mini_cluster.cc:1428] Started /tmp/dist-test-task05EAH3/build/debug/bin/kudu as pid 7747
I20260429 20:02:10.411441 27719 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task05EAH3/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1777492770945061-27719-0/minicluster-data/ts-1/wal/instance
I20260429 20:02:10.411902 27719 external_mini_cluster.cc:1468] Setting key b8c2293569735677fad376942c98ee08
W20260429 20:02:10.459610 6748 connection.cc:570] client connection to 127.27.17.254:46305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260429 20:02:10.460438 7889 negotiation.cc:336] Failed RPC negotiation. Trace:
0429 20:02:10.412472 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:33953)
0429 20:02:10.413096 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:10.413147 (+ 51us) client_negotiation.cc:175] Beginning negotiation
0429 20:02:10.413913 (+ 766us) client_negotiation.cc:253] Sending NEGOTIATE NegotiatePB request
0429 20:02:10.459690 (+ 45777us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":145,"threads_started":1}
W20260429 20:02:10.460896 7887 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:46305 (0 consecutive failures): Network error: Failed to ping master at 127.27.17.254:46305: Client connection negotiation failed: client connection to 127.27.17.254:46305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260429 20:02:10.461619 7889 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:10.461360 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:48553)
0429 20:02:10.461494 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:10.461559 (+ 65us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260429 20:02:10.462327 7889 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:10.462125 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:45599)
0429 20:02:10.462230 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:10.462272 (+ 42us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260429 20:02:10.462584 7887 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260429 20:02:11.087040 7738 heartbeater.cc:499] Master 127.27.17.254:46305 was elected leader, sending a full tablet report...
I20260429 20:02:11.088109 7891 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:11.087623 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.193:39959)
0429 20:02:11.087918 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:11.087988 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
W20260429 20:02:11.088541 7738 heartbeater.cc:646] Failed to heartbeat to 127.27.17.254:46305 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
I20260429 20:02:11.463887 7892 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:11.463410 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:33451)
0429 20:02:11.463718 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:11.463774 (+ 56us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260429 20:02:12.089840 7893 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:12.089308 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.193:44523)
0429 20:02:12.089669 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:12.089732 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
I20260429 20:02:12.465401 7894 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:12.464866 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:54959)
0429 20:02:12.465211 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:12.465283 (+ 72us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":80,"threads_started":1}
I20260429 20:02:13.091382 7895 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:13.090856 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.193:38969)
0429 20:02:13.091215 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:13.091273 (+ 58us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":170,"threads_started":1}
I20260429 20:02:13.467236 7896 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:13.466574 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:50315)
0429 20:02:13.467070 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:13.467132 (+ 62us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":86,"threads_started":1}
I20260429 20:02:14.092816 7897 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:14.092352 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.193:46243)
0429 20:02:14.092658 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:14.092717 (+ 59us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":100,"threads_started":1}
I20260429 20:02:14.468789 7898 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:14.468273 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:47523)
0429 20:02:14.468616 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:14.468679 (+ 63us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
I20260429 20:02:15.094434 7899 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:15.093910 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.193:57751)
0429 20:02:15.094250 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:15.094320 (+ 70us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":90,"threads_started":1}
I20260429 20:02:15.470436 7900 negotiation.cc:338] RPC negotiation tracing enabled. Trace:
0429 20:02:15.469923 (+ 0us) reactor.cc:678] Submitting negotiation task for client connection to 127.27.17.254:46305 (local address 127.27.17.194:42969)
0429 20:02:15.470234 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0429 20:02:15.470333 (+ 99us) negotiation.cc:326] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.17.254:46305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
F20260429 20:02:15.508028 27719 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1777492935 (unix time) try "date -d @1777492935" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800006c47) received by PID 27719 (TID 0x7ffba77a1d80) from PID 27719; stack trace: ***
@ 0x7ffbb3433980 (unknown) at ??:0
@ 0x7ffbb0a85e87 gsignal at ??:0
@ 0x7ffbb0a877f1 abort at ??:0
@ 0x7ffbb1cf2dcd google::LogMessage::Fail() at ??:0
@ 0x7ffbb1cf6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffbb1cf27cc google::LogMessage::Flush() at ??:0
@ 0x7ffbb1cf3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffbb4a7543b kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7ffbb4a72156 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x557a64f767bf kudu::SecurityITest::StartCluster() at ??:0
@ 0x557a64f5e622 kudu::SecurityITest_TestEncryptionWithKMSIntegration_Test::TestBody() at ??:0
@ 0x7ffbb2cfe0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7ffbb2cf2bdb testing::Test::Run() at ??:0
@ 0x7ffbb2cf2d9d testing::TestInfo::Run() at ??:0
@ 0x7ffbb2cf3377 testing::TestSuite::Run() at ??:0
@ 0x7ffbb2cf377c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7ffbb2cfe60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7ffbb2cf2e63 testing::UnitTest::Run() at ??:0
@ 0x557a64faeb0e RUN_ALL_TESTS() at ??:0
@ 0x557a64fadcfb main at ??:0
@ 0x7ffbb0a68c87 __libc_start_main at ??:0
@ 0x557a64f45efa _start at ??:0
2026-04-29 20:02:15.715 UTC [7453] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 20:02:15.716 UTC [7303] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 20:02:15.717 UTC [7299] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 20:02:15.717 UTC [7304] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 20:02:15.717 UTC [7301] FATAL: terminating connection due to unexpected postmaster exit
2026-04-29 20:02:15.719 UTC [7300] FATAL: terminating connection due to unexpected postmaster exit